6 Replies Latest reply: Sep 10, 2018 12:51 PM by rbfx rbfx RSS

    Question: Bigquery Incremental load

    rbfx rbfx

      Hi,

       

      I tried to execute incremental load from BigQuery, but looks like it's not working.

      Please advice.

      Information which I used to implement it: Basic Incremental Load - Qlik Sense - YouTube

       

      1 - Doesn't work:

       

      // get last updated row

      MaxKeyLoad:

      Load MAX(time) as MaxID

      FROM [lib://BI Qlik/BQ.qvd](qvd);

      LET MaxID = Peek('MaxID',0,MaxKeyLoad);

      // MaxID = 1530392398

       

      BQ:

      LOAD

          userId,

          clientId,

          date,

          time;

      SELECT

          userId,

          clientId,

          date,

          time;

      FROM `Streaming*`

      WHERE time > '$(MaxID)';

       

      Concatenate

      LOAD * FROM [lib://BI Qlik/BQ.qvd](qvd);

      STORE BQ into [lib://BI Qlik/BQ.qvd](qvd);

       

      2 - Get data from BigQuery without incremental load is working ok.

       

      // get last updated row

      MaxKeyLoad:

      Load MAX(time) as MaxID

      FROM [lib://BI Qlik/BQ.qvd](qvd);

      LET MaxID = Peek('MaxID',0,MaxKeyLoad);

      // MaxID = 1530392398

       

      BQ:

      LOAD

          userId,

          clientId,

          date,

          time;

      SELECT

          userId,

          clientId,

          date,

          time;

      FROM `Streaming*`

      WHERE time > 1530392398;

       

      Concatenate

      LOAD * FROM [lib://BI Qlik/BQ.qvd](qvd);

      STORE BQ into [lib://BI Qlik/BQ.qvd](qvd);