HiveThrift2 ACID Transactions?
Hi all, We want to use apply INSERTS, UPDATE, and DELETE operations on tables based on parquet or ORC files served by thrift2. Actually its unclear whether we can enable them and where. At the moment, when executing UPDATE or DELETE operations those are getting blocked. Anyone out who uses ACID transactions in combination with thrift2? Best, Meikel
Re: Log4J 2 Support
Hi Sean, I have had a more detailed look at what Spark is doing with log4 APIs and at this point I suspect that a logj 2.x migration might be more appropriate at the code level. That still does not solve the libraries issue though. That would need more investigation. I could be tempted to tackle it if there is enough interest. Cheers, Steve C On 10 Nov 2021, at 9:42 am, Sean Owen mailto:sro...@gmail.com>> wrote: Yep that's what I tried, roughly - there is an old jira about it. The issue is that Spark does need to configure some concrete logging framework in a few cases, as do other libs, and that isn't what the shims cover. Could be possible now or with more cleverness but the simple thing didn't work out IIRC. On Tue, Nov 9, 2021, 4:32 PM Stephen Coy mailto:s...@infomedia.com.au>> wrote: Hi there, It’s true that the preponderance of log4j 1.2.x in many existing live projects is kind of a pain in the butt. But there is a solution. 1. Migrate all Spark code to use slf4j APIs; 2. Exclude log4j 1.2.x from any dependencies sucking it in; 3. Include the log4j-over-slf4j bridge jar and slf4j-api jars; 4. Choose your favourite modern logging implementation and add it as a “runtime" dependency together with it’s slf4j binding jar (if needed). In fact in the short term you can replace steps 1 and 2 with "remove the log4j 1.2.17 jar from the distribution" and it should still work. The slf4j project also includes a commons-logging shim for capturing its output too. FWIW, the slf4j project is run by one of the original log4j developers. Cheers, Steve C On 9 Nov 2021, at 11:11 pm, Sean Owen mailto:sro...@gmail.com>> wrote: No plans that I know of. It's not that Spark uses it so much as its dependencies. I tried and failed to upgrade it a couple years ago. you are welcome to try, and open a PR if successful. On Tue, Nov 9, 2021 at 6:09 AM Ajay Kumar mailto:ajay.praja...@gmail.com>> wrote: Hi Team, We wanted to send Spark executor logs to a centralized logging server using TCP Socket. I see that the spark log4j version is very old(1.2.17) and it does not support JSON logs over tcp sockets on containers. I wanted to konw what is the plan for upgrading the log4j version to log4j2. Thanks in advance. Regards, Ajay This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/
Re: Log4J 2 Support
Yep that's what I tried, roughly - there is an old jira about it. The issue is that Spark does need to configure some concrete logging framework in a few cases, as do other libs, and that isn't what the shims cover. Could be possible now or with more cleverness but the simple thing didn't work out IIRC. On Tue, Nov 9, 2021, 4:32 PM Stephen Coy wrote: > Hi there, > > It’s true that the preponderance of log4j 1.2.x in many existing live > projects is kind of a pain in the butt. > > But there is a solution. > > 1. Migrate all Spark code to use slf4j APIs; > > 2. Exclude log4j 1.2.x from any dependencies sucking it in; > > 3. Include the log4j-over-slf4j bridge jar and slf4j-api jars; > > 4. Choose your favourite modern logging implementation and add it as a > “runtime" dependency together with it’s slf4j binding jar (if needed). > > In fact in the short term you can replace steps 1 and 2 with "remove the > log4j 1.2.17 jar from the distribution" and it should still work. > > The slf4j project also includes a commons-logging shim for capturing its > output too. > > FWIW, the slf4j project is run by one of the original log4j developers. > > Cheers, > > Steve C > > > On 9 Nov 2021, at 11:11 pm, Sean Owen wrote: > > No plans that I know of. It's not that Spark uses it so much as its > dependencies. I tried and failed to upgrade it a couple years ago. you are > welcome to try, and open a PR if successful. > > On Tue, Nov 9, 2021 at 6:09 AM Ajay Kumar wrote: > >> Hi Team, >> We wanted to send Spark executor logs to a centralized logging server >> using TCP Socket. I see that the spark log4j version is very old(1.2.17) >> and it does not support JSON logs over tcp sockets on containers. >> I wanted to konw what is the plan for upgrading the log4j version to >> log4j2. >> Thanks in advance. >> Regards, >> Ajay >> > > This email contains confidential information of and is the copyright of > Infomedia. It must not be forwarded, amended or disclosed without consent > of the sender. If you received this message by mistake, please advise the > sender and delete all copies. Security of transmission on the internet > cannot be guaranteed, could be infected, intercepted, or corrupted and you > should ensure you have suitable antivirus protection in place. By sending > us your or any third party personal details, you consent to (or confirm you > have obtained consent from such third parties) to Infomedia’s privacy > policy. http://www.infomedia.com.au/privacy-policy/ >
Re: Log4J 2 Support
Hi there, It’s true that the preponderance of log4j 1.2.x in many existing live projects is kind of a pain in the butt. But there is a solution. 1. Migrate all Spark code to use slf4j APIs; 2. Exclude log4j 1.2.x from any dependencies sucking it in; 3. Include the log4j-over-slf4j bridge jar and slf4j-api jars; 4. Choose your favourite modern logging implementation and add it as a “runtime" dependency together with it’s slf4j binding jar (if needed). In fact in the short term you can replace steps 1 and 2 with "remove the log4j 1.2.17 jar from the distribution" and it should still work. The slf4j project also includes a commons-logging shim for capturing its output too. FWIW, the slf4j project is run by one of the original log4j developers. Cheers, Steve C On 9 Nov 2021, at 11:11 pm, Sean Owen mailto:sro...@gmail.com>> wrote: No plans that I know of. It's not that Spark uses it so much as its dependencies. I tried and failed to upgrade it a couple years ago. you are welcome to try, and open a PR if successful. On Tue, Nov 9, 2021 at 6:09 AM Ajay Kumar mailto:ajay.praja...@gmail.com>> wrote: Hi Team, We wanted to send Spark executor logs to a centralized logging server using TCP Socket. I see that the spark log4j version is very old(1.2.17) and it does not support JSON logs over tcp sockets on containers. I wanted to konw what is the plan for upgrading the log4j version to log4j2. Thanks in advance. Regards, Ajay This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/
Re: Log4J 2 Support
No plans that I know of. It's not that Spark uses it so much as its dependencies. I tried and failed to upgrade it a couple years ago. you are welcome to try, and open a PR if successful. On Tue, Nov 9, 2021 at 6:09 AM Ajay Kumar wrote: > Hi Team, > We wanted to send Spark executor logs to a centralized logging server > using TCP Socket. I see that the spark log4j version is very old(1.2.17) > and it does not support JSON logs over tcp sockets on containers. > I wanted to konw what is the plan for upgrading the log4j version to > log4j2. > Thanks in advance. > Regards, > Ajay >
Log4J 2 Support
Hi Team, We wanted to send Spark executor logs to a centralized logging server using TCP Socket. I see that the spark log4j version is very old(1.2.17) and it does not support JSON logs over tcp sockets on containers. I wanted to konw what is the plan for upgrading the log4j version to log4j2. Thanks in advance. Regards, Ajay
