Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

hi Spark Team, I have raised a question on Spark through Stackoverflow. When you get a chance, can you please take a look and help me ?. https://stackoverflow.com/q/71431757/5927843 Thanks, Venkat 2016173438

Re: [SPARK-38438] pyspark - how to update spark.jars.packages on existing default context?

I don't know why I don't see my last message in the thread here: https://lists.apache.org/thread/5wgdqp746nj4f6ovdl42rt82wc8ltkcn Also don't get messages from Artemis in my mail, I can only see them in the thread web UI, which is very confusing. On top of that when I click on "reply via your own

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

Thank you Mich Talebzadeh for your answer. It's good to know that VARCHAR and CHAR are properly showing in Spark 3. Do you know what changed in Spark 3 that made this possible?. Or how can I achieve the same output in Spark 2.4.1? If there are some conf options, that would be helpful. Thanks,

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

Hive 3.1.1 Spark 3.1.1 Your stack overflow issue raised and I quote: "I have a need to generate DDL statements for Hive tables & views programmatically. I tried using Spark and Beeline for this task. Beeline takes around 5-10 seconds for each of the statements whereas Spark completes the same

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

Thank you. I am trying to get the table definition for the existing tables. BTW, the create and show command that you executed, was it on Spark 3.x ? . Thanks, Venkat 2016173438 On Fri, Mar 11, 2022 at 4:28 PM Mich Talebzadeh wrote: > Well I do not know what has changed. However, this should

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

ok. I work for an org where such upgrades take a few months. Not an immediate task. Thanks, Venkat 2016173438 On Fri, Mar 11, 2022 at 4:38 PM Mich Talebzadeh wrote: > yes in spark 3.1.1. Best to upgrade it to spark 3+. > > > >view my Linkedin profile >

unsubscribe

smime.p7s Description: S/MIME cryptographic signature

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

yes in spark 3.1.1. Best to upgrade it to spark 3+. view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of

StructuredStreaming - processing data based on new events in Kafka topic

Hello All, I have a structured Streaming program, which reads data from Kafka topic, and does some processing, and finally puts data into target Kafka Topic. Note : the processing is done in function - convertToDictForEachBatch(), which is called using - foreachBatch(convertToDictForEachBatch)

Re: Show create table on a Hive Table in Spark SQL - Treats CHAR, VARCHAR as STRING

Well I do not know what has changed. However, this should not affect your work. Try to create table in Spark sqltext: String = CREATE TABLE if not exists test.etcs( ID INT , CLUSTERED INT , SCATTERED INT , RANDOMISED INT , RANDOM_STRING VARCHAR(50) , SMALL_VC

Re: [SPARK-38438] pyspark - how to update spark.jars.packages on existing default context?

OK, I see the confusions in terminologies.  However, what were suggested should still work.  A Luigi worker in this case would function like a Spark client, responsible for submitting a Spark application (or job in Luigi's term).  In other words, you just define all necessary jars for all your

Unsubscribe

Re: Unsubscribe

please send an empty email to: user-unsubscr...@spark.apache.org to unsubscribe yourself from the list. On Sat, Mar 12, 2022 at 2:42 PM Aziret Satybaldiev < satybaldiev.azi...@gmail.com> wrote: >