date:20240519

Re: Windows Installer

2024-05-19 Thread Jialin Qiao

Hi,

That's Great!

If adding this installer, how much will our distribution package increase?

Jialin Qiao

Wang Critas  于2024年5月20日周一 09:57写道：
>
> Hi Trevor,
>
> So cool.
>
> Perhaps you can integrate it into iotdb/distribution. that would be more great
>
> Xuan Wang
>
> 发件人: Trevor Hart 
> 日期: 星期一, 2024年5月20日 09:48
> 收件人: Dev 
> 主题: Windows Installer
> Hello
>
>
>
> I dont know if there is some interest in this but I have built a Windows 
> installer for installing IoTDB on Windows servers.
>
>
>
> The installer includes Open JDK 11 and also does the following;
>
>
>
> 1. Creates a Windows service for the Config Node
>
> 2. Creates a Windows service for the Data Node
>
> 3. Open port 6667 on Windows firewall
>
>
>
> You can see the installer in action here; 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6BzCd-vAiGc%26t%3D1s%26ab_channel%3DOpeLtd=05%7C02%7C%7Cc4c099493f3e49494ae008dc786ef772%7C84df9e7fe9f640afb435%7C1%7C0%7C638517665208767354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=seKLvYsiEkBQsjdw6KFaIIUvL1%2FFtv7G36VQC2HVyb8%3D=0
>
>
>
> If there is any interest in this I can make it available.
>
>
> Thanks
>
> Trevor Hart

Re: Handling Duplicate Timestamps

2024-05-19 Thread Jialin Qiao

Hi Trevor,

Yes, IoTDB cannot handle this scenario currently because our primary
key is Path + Timestamp.

This year we will focus on the table model, a lot work to do :-)

Jialin Qiao

Trevor Hart  于2024年5月20日周一 09:45写道：
>
> Hi Jialin
>
>
>
> Yes the values would be different.
>
>
>
> As as example, these are from a web server log. The device is openzweb01 
> which is an IIS web server which may handle multiple requests at the same 
> time. The rows are unique in their own right but the timestamp is the same in 
> the logging.
>
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5
>
>
> 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6
>
> 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6
>
>
>
> This particular log file only records in seconds. So what I am doing with 
> these rows at the moment is to add an artitifical millisecond to enforce 
> uniqueness.
>
>
>
>
> 2024-05-20 00:00:14.000 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7
>
> 2024-05-20 00:00:14.001 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6
>
> 2024-05-20 00:00:14.002 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6
>
> 2024-05-20 00:00:14.003 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5
>
> 2024-05-20 00:00:15.000 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6
>
> 2024-05-20 00:00:15.001 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6
>
>
>
> For some other log files that I am processing they are in milliseconds 
> already but there is a (small) chance of dataloss if multiple requests happen 
> to be processed at the exact same time.
>
>
>
> I have been thinking about this some more and I think that rather than break 
> the IoTDB CRUD model I should handle this on the client side. In my use case 
> the log data is actually staged in an H2 database before it is sent to IoTDB 
> so I can enforce PK validation there. That way it is less expensive that 
> checking the timestamp in IoTDB for each record.
>
>
>
> Thanks
>
> Trevor Hart
>
>
>
>
>
>
>
>
>  On Fri, 17 May 2024 19:11:13 +1200 Jialin Qiao  
> wrote ---
>
>
>
> Hi Trevor,
>
> Will different values of the same timestamp be the same？
>
> 1. Same
> Time, Value
> 1, 1
> 1, 1
> 1, 1
>
> 2. Different
> Time, Value
> 1, 1
> 1, 2
> 1, 1
>
>
> Jialin Qiao
>
> Trevor Hart  于2024年5月14日周二 11:20写道：
> >
> > Thank you! I will implment some work around for now.
> >
> >
> > I would appreciate some consideration for this option in the future.
> >
> >
> > Thanks
> >
> > Trevor Hart
> >
> > Ope Limited
> >
> > w: http://www.ope.nz/
> >
> > m: +64212728039
> >
> >
> >
> >
> >
> >
> >
> >
> >  On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang 
> >  wrote ---
> >
> >
> >
> > > 1. Checking before insert if the timestamp already exists and remedy on 
> > > the client before resend
> > > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > > keep timestamp values unique.
> > Yes these maybe the best solutions for a specific application.
> >
> >
> > Analysis for IoTDB:
> > - Rejecting the write when receiving an existing timestamp in IoTDB is
> > time-costly (IoTDB needs to check historical data). I think we will do
> > not check it until we find a low-latency method.
> > - Allowing multiple value versions for a timestamp may introduce a
> > chain reaction and there may be a lot of codes that should be
> > modified, which is a huge work.
> >
> > There is a new idea (but I have no time to implement it...)
> > - Add a parameter in IoTDB: replace_strategy: first, last, avg etc...
> > - when an existing timestamp arrives, IoTDB accepts it
> > - when IoTDB runs LSM to merge data and meets multiple values for a
> > timestamp, then handles it according to the replace_startegy.
> >
> > The solution may also introduce some work to do... and we need to
> > think carefully the impact to the query process.
> > Need to survey whether this is a common requirement.
> >
> > Best,
> > ---
> > Xiangdong Huang
> >
> > Trevor Hart

答复: Windows Installer

2024-05-19 Thread Wang Critas

Hi Trevor,

So cool.

Perhaps you can integrate it into iotdb/distribution. that would be more great

Xuan Wang

发件人: Trevor Hart 
日期: 星期一, 2024年5月20日 09:48
收件人: Dev 
主题: Windows Installer
Hello



I dont know if there is some interest in this but I have built a Windows 
installer for installing IoTDB on Windows servers.



The installer includes Open JDK 11 and also does the following;



1. Creates a Windows service for the Config Node

2. Creates a Windows service for the Data Node

3. Open port 6667 on Windows firewall



You can see the installer in action here; 
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6BzCd-vAiGc%26t%3D1s%26ab_channel%3DOpeLtd=05%7C02%7C%7Cc4c099493f3e49494ae008dc786ef772%7C84df9e7fe9f640afb435%7C1%7C0%7C638517665208767354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=seKLvYsiEkBQsjdw6KFaIIUvL1%2FFtv7G36VQC2HVyb8%3D=0



If there is any interest in this I can make it available.


Thanks

Trevor Hart

Windows Installer

2024-05-19 Thread Trevor Hart

Hello



I dont know if there is some interest in this but I have built a Windows 
installer for installing IoTDB on Windows servers.



The installer includes Open JDK 11 and also does the following;



1. Creates a Windows service for the Config Node

2. Creates a Windows service for the Data Node

3. Open port 6667 on Windows firewall



You can see the installer in action here; 
https://www.youtube.com/watch?v=6BzCd-vAiGc=1s_channel=OpeLtd



If there is any interest in this I can make it available.


Thanks

Trevor Hart

Re: Handling Duplicate Timestamps

2024-05-19 Thread Trevor Hart

Hi Jialin



Yes the values would be different.



As as example, these are from a web server log. The device is openzweb01 which 
is an IIS web server which may handle multiple requests at the same time. The 
rows are unique in their own right but the timestamp is the same in the 
logging. 



2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7


2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6


2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6

2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5


2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6

2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6



This particular log file only records in seconds. So what I am doing with these 
rows at the moment is to add an artitifical millisecond to enforce uniqueness.




2024-05-20 00:00:14.000 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7 

2024-05-20 00:00:14.001 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6 

2024-05-20 00:00:14.002 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6

2024-05-20 00:00:14.003 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5 

2024-05-20 00:00:15.000 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6

2024-05-20 00:00:15.001 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6



For some other log files that I am processing they are in milliseconds already 
but there is a (small) chance of dataloss if multiple requests happen to be 
processed at the exact same time.



I have been thinking about this some more and I think that rather than break 
the IoTDB CRUD model I should handle this on the client side. In my use case 
the log data is actually staged in an H2 database before it is sent to IoTDB so 
I can enforce PK validation there. That way it is less expensive that checking 
the timestamp in IoTDB for each record.



Thanks 

Trevor Hart








 On Fri, 17 May 2024 19:11:13 +1200 Jialin Qiao  
wrote ---



Hi Trevor, 
 
Will different values of the same timestamp be the same？ 
 
1. Same 
Time, Value 
1, 1 
1, 1 
1, 1 
 
2. Different 
Time, Value 
1, 1 
1, 2 
1, 1 
 
 
Jialin Qiao 
 
Trevor Hart  于2024年5月14日周二 11:20写道： 
> 
> Thank you! I will implment some work around for now. 
> 
> 
> I would appreciate some consideration for this option in the future. 
> 
> 
> Thanks 
> 
> Trevor Hart 
> 
> Ope Limited 
> 
> w: http://www.ope.nz/ 
> 
> m: +64212728039 
> 
> 
> 
> 
> 
> 
> 
> 
>  On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang 
>  wrote --- 
> 
> 
> 
> > 1. Checking before insert if the timestamp already exists and remedy on the 
> > client before resend 
> > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > keep timestamp values unique. 
> Yes these maybe the best solutions for a specific application. 
> 
> 
> Analysis for IoTDB: 
> - Rejecting the write when receiving an existing timestamp in IoTDB is 
> time-costly (IoTDB needs to check historical data). I think we will do 
> not check it until we find a low-latency method. 
> - Allowing multiple value versions for a timestamp may introduce a 
> chain reaction and there may be a lot of codes that should be 
> modified, which is a huge work. 
> 
> There is a new idea (but I have no time to implement it...) 
> - Add a parameter in IoTDB: replace_strategy: first, last, avg etc... 
> - when an existing timestamp arrives, IoTDB accepts it 
> - when IoTDB runs LSM to merge data and meets multiple values for a 
> timestamp, then handles it according to the replace_startegy. 
> 
> The solution may also introduce some work to do... and we need to 
> think carefully the impact to the query process. 
> Need to survey whether this is a common requirement. 
> 
> Best, 
> --- 
> Xiangdong Huang 
> 
> Trevor Hart  于2024年5月14日周二 09:55写道： 
> > 
> > Hello Yuan 
> > 
> > 
> > 
> > Correct, the first timestamp and values should be retained. 
> > 
> > 
> > 
> > I realise this is does not align with the current design. I was just asking 
> > whether there was an existing option to operate to block duplicates. 
> > 
> > 
> > 
> > In a normal RDBMS if you try to insert with a duplicate the insert will 
> > fail with a PK violation. It would be great in some

Re: AW: Possible fix for randomly failing Jenkins builds

2024-05-19 Thread Haonan Hou

Approved and merged. 
Thanks.

Haonan

On 2024/05/17 10:14:10 Christofer Dutz wrote:
> Done:
> https://github.com/apache/iotdb/pull/12542
> Waiting for approval.
> 
> Chris
> 
> Von: Jialin Qiao 
> Datum: Freitag, 17. Mai 2024 um 09:15
> An: dev@iotdb.apache.org 
> Betreff: Re: Possible fix for randomly failing Jenkins builds
> +1 for importing the solutions in PLC4X
> 
> 
> Jialin Qiao
> 
> Christofer Dutz  于2024年5月14日周二 23:32写道：
> >
> > Hi all,
> >
> > I know that in IoTDB as well as PLC4X we were having issues of regularly 
> > but randomly failing Jenkins builds.
> >
> > Usually, these problems were because of timeouts when deploying to Nexus.
> >
> > In PLC4X I have now tried something that seems to be doing the job … 
> > possibly this could also help us here (I mean … the biggest part of the 
> > Jenkins setup is borrowed from PLC4X anyway ;-) )
> >
> > My solution was to wrap the execution of the deploy step in a loop on bash 
> > level:
> >
> > https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167
> >
> > sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
> > wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'
> >
> > So, it would try up to 5 times, if deployment fails.
> >
> > I think in case of deployment the full build is done and was successful. 
> > The only reason for failures in this step usually is problems with Nexus 
> > and a stupid retry should help.
> >
> > What do you think? Should we try the same change?
> >
> >
> > Chris
>

Re: Windows Installer

Re: Handling Duplicate Timestamps

答复: Windows Installer

Windows Installer

Re: Handling Duplicate Timestamps

Re: AW: Possible fix for randomly failing Jenkins builds

6 matches

Site Navigation

Mail list logo

Footer information