Re: Removing "fabric" from Ignite binary package name
Thats all for Anton only, my role not worth mentioning. > On 4 Aug 2018, at 08:12, Denis Magda wrote: > > Anton, Peter, thanks a lot! We all owe you a jar of beer and chocolate (?) > ;) Send me an invoice and I'll pay for it :) > > -- > Denis > > On Fri, Aug 3, 2018 at 7:32 AM Petr Ivanov wrote: > >> Yeap. When PR will be ready, I’ll start updating build configurations. >> >> >>> On 3 Aug 2018, at 17:28, Anton Vinogradov wrote: >>> >>> I can perform proper refactoring by myself. >>> >>> Peter, could you assist me with TC in that case? >>> >>> пт, 3 авг. 2018 г. в 17:17, Dmitriy Setrakyan : >>> Agree with Denis. Let's do the simple refactoring first, and then more complicated one in phase 2. On Fri, Aug 3, 2018 at 6:47 AM, Denis Magda wrote: >> >> Thus, my suggestion is: >> — find and update all hardcoded “fabric” usage on TC, so that they work >> both with or without fabric in name of binaries >> — use current implementation — review and merge to master >> — plan full suffix refactoring (as Anton suggests) on the next > iterations >> with no rush > > > I like this plan which allows us to do the things. I see the current > implementation as one of the steps of the overall refactoring proposed >> by > Anton. It sounds normal if we split refactoring into pieces. > > Anton, do you have time to help with the rest of refactoring tasks >> before > AI 2.7? It's great if we close the whole tasks by that time. Otherwise, > let's split it and release the current implementation first. > > -- > Denis > > On Fri, Aug 3, 2018 at 1:18 AM Petr Ivanov >> wrote: > >> Dmitriy, >> >> I cannot forecast estimates for this task as it dependents on many > factors: >> — I will be able to start researching the Anton’s implementation >> suggestion not earlier than the beginning of September >> — I am not acquainted with assembly configuration well, it may take some >> considerable time to understand how correctly get rid of “fabric” not >> touching everything else >> — the process of review and merge can also drag on indefinitely (based > on >> the previous attempt to introduce this changes) >> >> >> Vladimir, >> >> If community will approve this hack, I’ll implement it. >> Yet it won’t resolve the problem of building from sources not on TC — the >> fabric will stay in names of binaries and folders inside. >> And it will add problems when the correct implementation will be >> introduced. >> >> >> >> Thus, my suggestion is: >> — find and update all hardcoded “fabric” usage on TC, so that they work >> both with or without fabric in name of binaries >> — use current implementation — review and merge to master >> — plan full suffix refactoring (as Anton suggests) on the next > iterations >> with no rush >> >> >> >>> On 3 Aug 2018, at 09:50, Vladimir Ozerov wrote: >>> >>> Folks, >>> >>> Can you please explain the problem with TC and artifacts? Can we just >>> rename final artifact at the end of a build phase just before signing, >> and >>> leave the rest TC infrastructure as is? >>> >>> On Fri, Aug 3, 2018 at 12:28 AM Dmitriy Setrakyan < > dsetrak...@apache.org >>> >>> wrote: >>> Anton, Petr, Thanks for your readiness to assist. Can this be done for 2.7 release? D. On Thu, Aug 2, 2018 at 1:32 AM, Anton Vinogradov > wrote: > What I see is that we spent almost a year discussing how to do this. > I'm pretty sure we had enough time to do everything properly. > > So, proposal is to stop this discussion and start refactoring. > > I do not see any pitfalls and ready to assist if necessary. > > чт, 2 авг. 2018 г. в 5:14, Dmitriy Setrakyan < dsetrak...@apache.org >> : > >> I vote to remove the fabric from the build in the easiest way >> possible. > Can >> other Igniters comment? >> >> D. >> >> On Wed, Aug 1, 2018 at 12:46 PM, Petr Ivanov > wrote: >> >>> My concern here is exactly about internal build processes — > removing >>> fabric from the name of binary archive (with any way) will break > lots > of >>> them. >>> There will be no sacrifices, just lots of work for fixing build > processes >>> (where we won’t be able to introduce changes proactively). >>> >>> Therefore only fabric removal implementation (quick with some > legacy > left >>> or full refactoring) is on the agenda. >>> And this matter should be j
Re: Ignite as distributed file storage
Dmitriy, This approach will also work with byte[] array and binary objects as well. It will be just new addition to binary object types, but behaviour will be same. What do you mean saying custom logic of splittiing blob? By default blob will be splitted to chunks of some size. This will be configurable parameter. The mechanism of uploading blobs will be different than we use in our cache API. E.g. we will upload blob to primary and backup nodes in parallel on the client side, instead of forwarding pieces of data from primary to backups. To operate with huge data and streaming data I don't think that our cache API will be good choice, as it doesn't have instruments to manage striped blobs (blob spreaded across several nodes) or streaming data (with undefined size). Most of the functionality in our cache API is inapplicable or unnecessary for big data manipulating. Instead of it I suggest to agree with Vladimir approach and try to refactor and use IGFS, if you want to operate only with BLOBs. As conclusion the whole storage implementation can be divided to several pieces: 1) Internal storage implementation + metadata cache over it. 2) Communication SPI for storage, upload/download sessions. 3) Applied blobs as part of binary objects, java objects, etc. 4) Binary/Web API to store BLOBs directly to storage and operations with them (pre/post processing, transformation, Map-Reduce). 5) Intergration with SQL Alexey, In systems that operates with huge data such concept as rebalance is hard to implement and there is no big impact of it. If cluster grows, metadata of the blobs will be rebalanced automatically using already implemented mechanisms. But data will remain on previous place. Mechanism of data rebalancing should be implemented separately and should be used manually. 2018-08-04 7:49 GMT+03:00 Denis Magda : > Dmitriy, > > I would suggest us not limiting blobs use case to a dedicated cache. If to > look at other databases they usually have BLOB/LONGBLOB/etc. as a type > meaning that users mix simple and BLOB types in the same tables. > > Should we start with Ignite SQL adding blobs through its APIs? > > -- > Denis > > On Fri, Aug 3, 2018 at 1:52 PM Dmitriy Setrakyan > wrote: > > > Pavel, > > > > Not everything that gets put in Ignite has a class, and not everything > can > > be annotated. For example, what about byte[] or a binary object? > > > > I would prefer a separate cache with specific purpose of storing blobs. > It > > will be easier for users to understand and configure. It can also have a > > custom logic of splitting a blob into multiple batches to be able to > > transfer it over the network faster. > > > > D. > > > > > > On Fri, Aug 3, 2018 at 3:21 AM, Pavel Kovalenko > > wrote: > > > > > Dmitriy, > > > > > > I think we don't need a separate implementation of cache like BLOB > cache. > > > Instead of it, a user can mark value class or value class field with > the > > > special annotation "@BLOB". > > > During cache put, marshaller will place a special placeholder on such > > > fields, write byte[] array payload of a field to special internal blob > > > storage and place the only reference to actual DataEntry in the page > > > memory. > > > During cache get, marshaller will place a special proxy instead of an > > > actual class that can be downloaded and unmarshalled by demand from > > > internal storage on the user side. > > > Using such approach we will also solve eager/lazy problem, this will > also > > > give user possibility to adjust his own marshallers (like Jackson, > Jaxb, > > > etc.) to marshal/unmarshal his blob classes from/to byte[] arrays. > > > No major changes in public API are required, it can be pluggable > > component. > > > > > > > > > 2018-08-03 0:25 GMT+03:00 Dmitriy Setrakyan : > > > > > > > On Thu, Aug 2, 2018 at 1:08 AM, Pavel Kovalenko > > > > wrote: > > > > > > > > > Dmitriy, > > > > > > > > > > I still don't understand why do you think that it will be file > > system? > > > > > In all my previous messages I emphasized that this storage > shouldn't > > be > > > > > considered as a file system. It's just a large data storage, whose > > > > entities > > > > > can be easily accessed using key/link (internally, or externally > > using > > > > > web/binary protocol interfaces). > > > > > > > > > > > Instead, if we must focus on large blobs, I would solve the > problem > > > of > > > > > supporting large blobs in regular Ignite caches, as I suggested > > before. > > > > > > > > > > This is impossible. Our page memory can't handle efficiently it by > > > > design. > > > > > > > > > > > > > But our API does. What is stopping us from creating a cache as a BLOB > > > cache > > > > and using whatever storage we need? > > > > > > > > > >
Re: [MTCGA]: new failures in builds [1575775] needs to be handled
Yakov, The short answer is - "Yes, we've missed it" and for the second "I don't know". Personally, I'm not supporting any merges to master with failing tests even with flaky. So, I did everething possible to catch all such tests in my local branch, but I've missed this case with LOCAL caches. This case was covered only by .NET test and wasn't covered by JUnit. Perhaps it happend partly because of we already had constanly failed tests like this one [1] or timeouts like [2] and it is really hard to determine what's goin on in .NET. I think it would be very helpfull addittion if we tune our MTCGA.Bot to catch such cases in local branches. I'll do my best to help community with it. [1] https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-4642953772602570726&tab=testDetails [2] https://ci.ignite.apache.org/viewLog.html?buildId=1544712&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_PlatformNetLongRunning On Fri, 3 Aug 2018 at 12:22 Yakov Zhdanov wrote: > Maxim, did we have .net tests failing before merge? If yes, how come we > merged it? > > --Yakov > > 2018-08-02 18:03 GMT+03:00 Maxim Muzafarov : > > > Folks, > > > > Seems like rebalancing changes leads us to hanging TestRebalance() test > for > > .NET. > > Created issue [1], I will try to investigate it. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9170 > > > > On Thu, 2 Aug 2018 at 17:12 wrote: > > > > > Hi Ignite Developer, > > > > > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be > addressed. > > > I hope you can help. > > > > > > *New Critical Failure in master Platform .NET (Long Running) > > > https://ci.ignite.apache.org/viewType.html?buildTypeId= > > IgniteTests24Java8_PlatformNetLongRunning&branch=%3Cdefault%3E&tab= > > buildTypeStatusDiv > > > Changes may led to failure were done by > > > - sboikov > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827278&personal=false > > > - saikat.maitra > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827276&personal=false > > > - maxmuzaf > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827268&personal=false > > > - biryukovvitaliy92 > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827267&personal=false > > > - alkuznetsov.sb > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827258&personal=false > > > - ilantukh > > > http://ci.ignite.apache.org/viewModification.html?modId= > > 827250&personal=false > > > > > > - If your changes can led to this failure(s), please create > issue > > > with label MakeTeamCityGreenAgain and assign it to you. > > > -- If you have fix, please set ticket to PA state and write to > > dev > > > list fix is ready > > > -- For case fix will require some time please mute test and set > > > label Muted_Test to issue > > > - If you know which change caused failure please contact change > > > author directly > > > - If you don't know which change caused failure please send > > > message to dev list to find out > > > Should you have any questions please contact dpav...@apache.org or > write > > > to dev.list > > > Best Regards, > > > MTCGA.Bot > > > Notification generated at Thu Aug 02 17:12:39 MSK 2018 > > > > > -- > > -- > > Maxim Muzafarov > > > -- -- Maxim Muzafarov
