I am sorry, I should have explained this sooner. When I do this:
(.getBytes document-as-string "UTF-8")
The quote marks are always escaped. When I do this:
document-as-byte-array (bytes (byte-array (map (comp byte int)
document-as-string)))
The quote marks are sometimes escaped, but most of the time they are not.
I need to avoid having those quote marks escaped. I am not clear what is
causing the escaping.
On Thursday, July 9, 2015 at 12:01:51 PM UTC-4, Francis Avila wrote:
>
> your document-as-byte-array is wrong--it only handles ascii, and very
> inefficiently too.
>
> Just say (.getBytes document-as-string "UTF-8") to get a utf-8 encoding of
> the string as a byte array.
>
> On Thu, Jul 9, 2015 at 10:41 AM, <gingers...@gmail.com <javascript:>>
> wrote:
>
>>
>> > As for the escaped quotes, you may be using pr or prn to print, or
>> maybe you are
>> > using pr-str to produce the string representation. I can't be sure.
>>
>> At the moment I create the string like this:
>>
>> document-as-string (str "{\"transaction-id\" : \"" transaction-id
>> "\", \"message\" : \"" message "\"}")
>> document-as-byte-array (bytes (byte-array (map (comp byte int)
>> document-as-string)))
>>
>> The crazy thing is that this sometimes works, but other times the quote
>> marks appear in Redis as escaped quote marks. As near as I can tell, the
>> important factor is the length of the string. A short string is likely to
>> have its quote marks escaped. A long string does not have its quote marks
>> escaped.
>>
>> My co-worker is working on the Java app. I am working on the Clojure app.
>> We can both adjust out apps freely, just so long as we can get data to and
>> from each other, in a manner that allows us to eventually cast the data to
>> and from JSON.
>>
>> Any suggestions are welcome.
>>
>>
>>
>>
>>
>>
>> On Wednesday, July 8, 2015 at 8:58:53 PM UTC-4, Francis Avila wrote:
>>>
>>> Who is saving these strings, and who is reading them? Do you have
>>> complete control over both apps, or does one of them need to be aligned
>>> with the other?
>>>
>>> If the Java app is the baseline, you need to know the exact details of
>>> the format of the data it saves. Just knowing "it's JSON" is not enough,
>>> because there may be other data types (e.g. dates) that don't have native
>>> JSON representations. (The go-to JSON de/encoder for clojure is cheshire:
>>> https://github.com/dakrone/cheshire )
>>>
>>> I took a quick look at Jedis and the easy, default way of using it is
>>> with strings. It will encode strings to UTF-8 before sending to Redis and
>>> decode from UTF-8 on read. You can set raw byte arrays too (which will not
>>> be altered in any way before sending), but it's not clear to me how it can
>>> read out raw byte arrays. (I'm sure there's a way, but it's not immediately
>>> obvious.)
>>>
>>> As for the escaped quotes, you may be using pr or prn to print, or maybe
>>> you are using pr-str to produce the string representation. I can't be sure.
>>>
>>> On Wednesday, July 8, 2015 at 5:31:27 PM UTC-5, gingers...@gmail.com
>>> wrote:
>>>>
>>>> And I have another stupid question. Using the above code, I am
>>>> sometimes getting strings in Redis that have escaped quotation marks, like
>>>> this:
>>>>
>>>> " \"transaction-id\" : \" 1ec47c2e-21ee-427c-841c-80a0f89f55d7 \"
>>>> \"debrief\" : \" Susan Hilly at Citi called to get a quotation for
>>>> discounted weekly car rental for approximately 3 cars per week, or 150
>>>> rentals annually. \" "
>>>>
>>>> Why is that happening?
>>>>
>>>>
>>>>
>>>> On Wednesday, July 8, 2015 at 5:38:20 PM UTC-4, gingers...@gmail.com
>>>> wrote:
>>>>>
>>>>> Francis Avila,
>>>>>
>>>>> Thank you for your response. The Java app is using Jedis and the
>>>>> Clojure app is using Carmine. I'm wondering if you can suggest what you
>>>>> think would be the easiest way to allow these 2 apps to understand each
>>>>> other's strings?
>>>>>
>>>>> You were correct about how unsafe the above code was. I tested it for
>>>>> less than 15 minutes and ran into the fact that a \n newline made a mess
>>>>> of
>>>>> everything.
>>>>>
>>>>>
>>>>>
>>>>> On Wednesday, July 8, 2015 at 5:15:35 PM UTC-4, Francis Avila wrote:
>>>>>>
>>>>>> You are running into Carmine's automatic nippy serialization.
>>>>>> https://github.com/ptaoussanis/carmine#serialization
>>>>>>
>>>>>> Redis only stores byte arrays (what it calls "strings"). Carmine uses
>>>>>> the nippy library (the meaning of "NPY" in your byte stream) to
>>>>>> represent
>>>>>> rich types compactly as bytes. https://github.com/ptaoussanis/nippy
>>>>>>
>>>>>> If you give Carmine a byte array to store, it will store it directly
>>>>>> without nippy-encoding it. E.g. (.getBytes "{}" "UTF-8")
>>>>>>
>>>>>> BTW your document-as-string example is extremely unsafe: how will you
>>>>>> reliably read this message out again? e.g. what if the 'debrief' string
>>>>>> contains a single quote? Use a proper serialization format.
>>>>>>
>>>>>> So the key is to have both your Clojure and Java app store *bytes* in
>>>>>> Redis using the same serialization. You can store anything you want
>>>>>> (nippy,
>>>>>> utf-8-encoded json, fressian, bson, utf-8 xml, utf-16 java strings,
>>>>>> whatever) as long as it's bytes and it's read and written the same way
>>>>>> in
>>>>>> all your apps.
>>>>>>
>>>>>> The Redis library your Java app is using may have its own automatic
>>>>>> de/serialization, too. You need to find out what it's doing and either
>>>>>> work
>>>>>> with this or turn it off, just like with Carmine.
>>>>>>
>>>>>> Nippy unfortunately does not have a Java API out of the box:
>>>>>> https://github.com/ptaoussanis/nippy/issues/66
>>>>>>
>>>>>>
>>>>>> On Wednesday, July 8, 2015 at 3:35:49 PM UTC-5, gingers...@gmail.com
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I am not sure if this is a Clojure question or a Java question. I
>>>>>>> don't know Java, so I could use whatever help folks can offer.
>>>>>>>
>>>>>>> "primitive string" here means what I can write when I am at the
>>>>>>> terminal.
>>>>>>>
>>>>>>> We have 2 apps, one in Clojure, one in Java. They talk to each other
>>>>>>> via Redis. I know the Java app can read stuff out of Redis, using our
>>>>>>> "transaction-id", if I use the terminal and open up "redis-clj" and
>>>>>>> write a
>>>>>>> string directly from the terminal. But I have this Clojure code, which
>>>>>>> depends on Peter Taoussanis's Carmine library:
>>>>>>> (defn worker [document]
>>>>>>> {:pre [(string? (:transaction-id document))]}
>>>>>>> (let [transaction-id (:transaction-id document)
>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>> redis-connection {:pool {} :spec {:host "127.0.0.1" :port
>>>>>>> 6379 }}]
>>>>>>> (timbre/log :trace " message we will send to NLP "
>>>>>>> document-as-string)
>>>>>>> (carmine/wcar redis-connection (carmine/set transaction-id
>>>>>>> document))
>>>>>>> (loop [document-in-redis (carmine/wcar redis-connection
>>>>>>> (carmine/get transaction-id))]
>>>>>>>
>>>>>>> (if-not (.contains (first document-in-redis) "processed")
>>>>>>> (recur (carmine/wcar redis-connection (carmine/get
>>>>>>> transaction-id)))
>>>>>>> (do
>>>>>>> (carmine/wcar redis-connection (carmine/del
>>>>>>> transaction-id))
>>>>>>> document-in-redis)))))
>>>>>>>
>>>>>>> This line in particular, I have tried doing this several ways:
>>>>>>>
>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>>
>>>>>>> In Redis, I expect to see:
>>>>>>>
>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>> responded
>>>>>>> to our proposal and said his company is read to move forward. The rate
>>>>>>> of
>>>>>>> $400 per ton of shredded paper was acceptable to them, and they shred
>>>>>>> about
>>>>>>> 2 tons of documents every month. $96,000 in potential revenue annually.
>>>>>>> I
>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>
>>>>>>> But if I then launch redis-cli, I see:
>>>>>>>
>>>>>>>
>>>>>>> 127.0.0.1:6379> keys *
>>>>>>> 1) "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>>
>>>>>>> 127.0.0.1:6379> get "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>> "\x00>NPY\b\x00\x00\x01\xfc\xf1\xfe\x1b\x00\x00\x00\nj\nip-addressi\x0e165.254.84.238j\x05tokeni$46b87d64-cff3-4b8b-895c-e089ac59544dj\x0bapi-versioni\x02v1j\x0etransaction-idi$42e574e7-3b80-424a-b9ff-01072f1e0358j\adebrief\r\x00\x00\x01YSmeek
>>>>>>>
>>>>>>> Hallie of Withers, Smeg, Harrington and Norvig responded to our
>>>>>>> proposal
>>>>>>> and said his company is rea-\x00\xf1\x06move forward. The
>>>>>>> raty\x00\xf0\x0c$400 per ton of shredded pa\x16\x00\xf1\bwas acceptable
>>>>>>> to
>>>>>>> them,q\x00Bthey0\x00\x80 about 2E\x00\x10sF\x00\xf1Ldocuments every
>>>>>>> month.
>>>>>>> $96,000 in potential revenue annually. I will meet with him
>>>>>>> tomorrow{\x00\"we#\x00@sign\x92\x00\xa0 contract."
>>>>>>>
>>>>>>>
>>>>>>> I don't know what all of those extra characters are. The Java app is
>>>>>>> not picking this item up, so I assume the Java app is not seeing this
>>>>>>> as a
>>>>>>> string. I expected this to look the same as if I had written this at
>>>>>>> the
>>>>>>> terminal:
>>>>>>>
>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>> responded
>>>>>>> to our proposal and said his company is read to move forward. The rate
>>>>>>> of
>>>>>>> $400 per ton of shredded paper was acceptable to them, and they shred
>>>>>>> about
>>>>>>> 2 tons of documents every month. $96,000 in potential revenue annually.
>>>>>>> I
>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>
>>>>>>> I assume it is easy to get a string into a format that can be
>>>>>>> understood by both a Clojure app and a Java app. I don't care what
>>>>>>> format
>>>>>>> that is, but it needs to be consistent.
>>>>>>>
>>>>>>> Can anyone make suggestions about what I can do to make sure the
>>>>>>> Clojure app and the Java app both write to Redis using a format that
>>>>>>> the
>>>>>>> other will understand? In particular, both apps need to see the
>>>>>>> "'transaction-id".
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Clojure" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/clojure/9swfBxAbT90/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
