Would it be possible to use the connector as a datasource in the dashboard for creating gadgets?
On Sat, Aug 15, 2015 at 9:47 AM, Inosh Goonewardena <in...@wso2.com> wrote: > Hi, > >> >> 1. Adding new spark dialects related for various dbs (WIP) >> >> I have added new spark JDBC dialects for following DBs. > > - mysql > - mssql > - oracle > - postgres > - db2 > > > No. Not incremental data processing. My question regarding the deleting >> entire summery table records and re-insert again. IMO, doing upsert will be >> more efficient than your above approach. Again, if there is no other >> option, above re-insert is done as a batch operation or are you insert >> record one by one? >> > > Yes. We had this implementation in Hive JDBC Handler. It is possible to > follow the similar approach here, i.e., if upsert queries are supported by > the DB(most DBs support merge queries) we can let the user to specify the > upsert query to be used in the spark script and if the upsert query is not > provided we can check the records exist in the table by using primary key > and update or insert the records accordingly. But, IMO, since we support > update operations in our data layer(when the CarbonAnalytics is used), we > shouldn't worry that much about supporting it in Carbon Spark JDBC > connector. On the other hand if that is how Spark supports insert > into/overwrite by default, we should follow the same approach. > > On Thu, Aug 13, 2015 at 8:18 AM, Gihan Anuruddha <gi...@wso2.com> wrote: > >> Hi Niranda, >> >> No. Not incremental data processing. My question regarding the deleting >> entire summery table records and re-insert again. IMO, doing upsert will be >> more efficient than your above approach. Again, if there is no other >> option, above re-insert is done as a batch operation or are you insert >> record one by one? >> >> Regards, >> Gihan >> >> On Wed, Aug 12, 2015 at 11:40 AM, Niranda Perera <nira...@wso2.com> >> wrote: >> >>> Hi Gihan, >>> >>> are we talking about incremental processing here? insert into/overwrite >>> queries will normally be used to push analyzed data into summary tables. >>> >>> in the spark jargon, insert overwrite table means, completely deleting >>> the table and recreating it. I'm a confused with the meaning of 'overwrite' >>> with respect to the previous 2.5.0 Hive scripts, are doing an update there? >>> >>> rgds >>> >>> On Tue, Aug 11, 2015 at 7:58 PM, Gihan Anuruddha <gi...@wso2.com> wrote: >>> >>>> Hi Niranda, >>>> >>>> Are we going to solve those limitations before the GA? Specially >>>> limitation no.2. Over time we can have stat table with thousands of >>>> records, so are we going to remove all the records and reinsert every time >>>> that spark script runs? >>>> >>>> Regards, >>>> Gihan >>>> >>>> On Tue, Aug 11, 2015 at 7:13 AM, Niranda Perera <nira...@wso2.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> we have implemented a custom Spark JDBC connector to be used in the >>>>> Carbon environment. >>>>> >>>>> this enables the following >>>>> >>>>> 1. Now, temporary tables can be created in the Spark environment >>>>> by specifying an analytics datasource (configured by the >>>>> analytics-datasources.xml) and a table >>>>> 2. Spark uses "SELECT 1 FROM $table LIMIT 1" query to check the >>>>> existence of a table and the LIMIT query is not provided by all dbs. >>>>> With >>>>> the new connector, this query can be provided with as a config. (this >>>>> config is still WIP) >>>>> 3. Adding new spark dialects related for various dbs (WIP) >>>>> >>>>> the idea is to test this for the following dbs >>>>> >>>>> - mysql >>>>> - h2 >>>>> - mssql >>>>> - oracle >>>>> - postgres >>>>> - db2 >>>>> >>>>> I have loosely tested the connector with MySQL, and I would like the >>>>> APIM team to use it with the API usage stats use-case, and provide us some >>>>> feedback. >>>>> >>>>> this connector can be accessed as follows. (docs are still not >>>>> updated. I will do that ASAP) >>>>> >>>>> create temporary table <temp_table> using CarbonJDBC options >>>>> (dataSource "<datasource name>", tableName "<table name>"); >>>>> >>>>> select * from <temp_table> >>>>> >>>>> insert into/overwrite table <temp_table> <some select statement> >>>>> >>>>> known limitations >>>>> >>>>> 1. when creating a temp table, it should already be created in >>>>> the underlying datasource >>>>> 2. "insert overwrite table" deletes the existing table and creates >>>>> it again >>>>> >>>>> >>>>> would be very grateful if you could use this connector in your current >>>>> JDBC use cases and provide us with feedback. >>>>> >>>>> best >>>>> -- >>>>> *Niranda Perera* >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94-71-554-8430 >>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>> https://pythagoreanscript.wordpress.com/ >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> architect...@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> W.G. Gihan Anuruddha >>>> Senior Software Engineer | WSO2, Inc. >>>> M: +94772272595 >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Niranda Perera* >>> Software Engineer, WSO2 Inc. >>> Mobile: +94-71-554-8430 >>> Twitter: @n1r44 <https://twitter.com/N1R44> >>> https://pythagoreanscript.wordpress.com/ >>> >> >> >> >> -- >> W.G. Gihan Anuruddha >> Senior Software Engineer | WSO2, Inc. >> M: +94772272595 >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > > _______________________________________________ > Architecture mailing list > architect...@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards Iranga Muthuthanthri (M) -0777-255773 Team Product Management
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
