Re: Support updating/deleting data for stream table
Since time lapses， the table will become bigger and bigger, we do need one way to clean data out of date. Make StreamTable support partition maybe a good choice. The first step we can make this simple, today's data will be mixed with normal segment and stream segment. when rolling, we will make non-today's partition normal segments. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Support updating/deleting data for stream table
Hi dev: Sometimes we need to delete some historical data from stream table to make the table size not too large, but currently the stream table can't support updating/deleting data, so we need to stop the app and use 'alter table COMPACT 'close_streaming' command to close stream table, and then delete data. According to discussion with Jacky and David offline, there are two solutions to resolve this without stopping app: 1. set all non-stream segments to 'carbon.input.segments.tablename' property to delete data except stream segment, this's easy to implement; 2. support deleting data for stream segment too, this's more complicated. I think we can implement with solution 1 first, and then consider the implementation of solution 2 in depth. Welcome to feedback, thanks. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: loading data from parquet table always
Hi, The exception says, there is problem while copying from local to carbonstore(HDFS). It means the writing has already finished in the temp folder and after writing it will copy the files to hdfs and it is failing during that time. So with this exception trace, it will be difficult to know the root cause for the failure, failure can happen because of HDFS also. So you can check two things 1. Check whether the space is available in HDFS or not 2. When this exception came, check what is the exception in hdfs logs. May be with that you can get some idea. There is one property called *carbon.load.directWriteHdfs.enabled* By default, this property will be false, and if you make it true, the files will be directly written to carbonstore(hdfs), instead of writing first in local and then copying. You can check by setting this property whether the load is successful or not. Regards, Akash R Nilugal -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: loading data from parquet table always
Hi, The exception says, there is problem while copying from local to carbonstore(HDFS). It means the writing has already finished in the temp folder and after writing it will copy the files to hdfs and it is failing during that time. So with this exception trace, it will be difficult to know the root cause for the failure, failure can happen because of HDFS also. So you can check two things 1. Check whether the space is available in HDFS or not 2. When this exception came, check what is the exception in hdfs logs. May be with that you can get some idea. There is one property called *carbon.load.directWriteHdfs.enabled* By default, this property will be false, and if you make it true, the files will be directly written to carbonstore(hdfs), instead of writing first in local and then copying. You can check by setting this property whether the load is successful or not. Regards, Akash R Nilugal -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
