Re: Save the date: ApacheCon North America, September 24-27 in Montréal

[Dominic Tarr]

unsubscribe

On Wed, Feb 21, 2018 at 3:21 AM, Rich Bowen  wrote:
> Dear Apache Enthusiast,
>
> (You’re receiving this message because you’re subscribed to a user@ or dev@
> list of one or more Apache Software Foundation projects.)
>
> We’re pleased to announce the upcoming ApacheCon [1] in Montréal, September
> 24-27. This event is all about you — the Apache project community.
>
> We’ll have four tracks of technical content this time, as well as lots of
> opportunities to connect with your project community, hack on the code, and
> learn about other related (and unrelated!) projects across the foundation.
>
> The Call For Papers (CFP) [2] and registration are now open. Register early
> to take advantage of the early bird prices and secure your place at the
> event hotel.
>
> Important dates
> March 30: CFP closes
> April 20: CFP notifications sent
> August 24: Hotel room block closes (please do not wait until the
> last minute)
>
> Follow @ApacheCon on Twitter to be the first to hear announcements about
> keynotes, the schedule, evening events, and everything you can expect to see
> at the event.
>
> See you in Montréal!
>
> Sincerely, Rich Bowen, V.P. Events,
> on behalf of the entire ApacheCon team
>
> [1] http://www.apachecon.com/acna18
> [2] https://cfp.apachecon.com/conference.html?apachecon-north-america-2018

Re: suggestion for improvement

[Dominic Tarr]

I'm working on some projects that may be of use to you.

I'm already down these rabbit holes!

I have https://github.com/dominictarr/xdiff which diffs and patches json.
it can also do a 3-way merge, which is used to reconcile concurrent changes.

Also, I have https://github.com/dominictarr/snob which manages series of
changes.
it's the same architecture as git. but allows you to plug in a difftool
(that implements diff, diff3, and patch)

since you have already architected your application, xdiff may be more
useful to you.
either way, i've been working on these mainly because they are interesting
problems,
and would be very interested to see how they apply in practice.

is the browser also sending back changes? or is it only going one way?

On Thu, Apr 19, 2012 at 2:56 PM, Mark Hahn m...@hahnca.com wrote:

 I'm sending changed docs from node to the browser.  Then the browser
 compares the doc to the dom.   If i only sent doc changes there would be
 less net traffic and several many of changes wouldn't need to be sent at
 all.

 The particular situation that caused me to post this question is a case
 where only about 1% of the changes are needed.  It bugs me to have so much
 unneeded traffic.

 I could cache all the data sent to the client to compare in the server.
  I've also thought of writing a summary of what is changed on writes along
 with the document.  I wanted to ask here before I went down any of these
 rabbit holes.

 On Wed, Apr 18, 2012 at 6:50 PM, Dominic Tarr dominic.t...@gmail.com
 wrote:

  couch just doesn't track things at that level, since versioning is done
 at
  the document level.
 
  how are you doing handling this at the moment?
 
  are you using a library to do a diff of the conflicted versions or
  something like that?
 
  On Thu, Apr 19, 2012 at 8:28 AM, Mark Hahn m...@hahnca.com wrote:
 
   Was there a suggestion in the voting list to tell the user what has
  changed
   when a change notification or conflict is issued?  My app burns a lot
 of
   resources because of this problem.
  
 

Re: suggestion for improvement

[Dominic Tarr]

I'm think mark wants to know what has changed with the last update,
rather than to keep every version around forever.

On Fri, Apr 20, 2012 at 10:10 AM, Robert Newson rnew...@apache.org wrote:

 You can fetch previous revisions with ?rev=rev until compaction
 sweeps them away, but it's considered bad practice (your administrator
 should be free to compact whenever needed).

 I don't think it's CouchDB's job to keep all the old versions of
 documents forever, this should be solved at the application layer, it
 doesn't seem particularly difficult either. If you care about the
 changes a doc goes through, you should record them in the doc itself.
 Whenever you update a document, add a new entry to a history array,
 for example. We have this for replication checkpoints already.

 B.

 On 19 April 2012 22:33, Mark Hahn m...@hahnca.com wrote:
  Thanks, I had forgotten about open_revs.  But open_revs only applies to
  conflicts, right?
 
  How do I find out what changed in a change feed?  In other words, if I
 get
  a change feed is there a way to access the previous version to find out
  what changed?

Re: suggestion for improvement

[Dominic Tarr]

you would need to respond to each item in the feed by making a request back
to couch
(since merely including the document is not sufficient, in your case)
and then combining that into your own augmented changes feed, perhaps.

On Fri, Apr 20, 2012 at 11:01 AM, Dominic Tarr dominic.t...@gmail.comwrote:

 I'm think mark wants to know what has changed with the last update,
 rather than to keep every version around forever.


 On Fri, Apr 20, 2012 at 10:10 AM, Robert Newson rnew...@apache.orgwrote:

 You can fetch previous revisions with ?rev=rev until compaction
 sweeps them away, but it's considered bad practice (your administrator
 should be free to compact whenever needed).

 I don't think it's CouchDB's job to keep all the old versions of
 documents forever, this should be solved at the application layer, it
 doesn't seem particularly difficult either. If you care about the
 changes a doc goes through, you should record them in the doc itself.
 Whenever you update a document, add a new entry to a history array,
 for example. We have this for replication checkpoints already.

 B.

 On 19 April 2012 22:33, Mark Hahn m...@hahnca.com wrote:
  Thanks, I had forgotten about open_revs.  But open_revs only applies to
  conflicts, right?
 
  How do I find out what changed in a change feed?  In other words, if I
 get
  a change feed is there a way to access the previous version to find out
  what changed?

Re: suggestion for improvement

[Dominic Tarr]

couch just doesn't track things at that level, since versioning is done at
the document level.

how are you doing handling this at the moment?

are you using a library to do a diff of the conflicted versions or
something like that?

On Thu, Apr 19, 2012 at 8:28 AM, Mark Hahn m...@hahnca.com wrote:

 Was there a suggestion in the voting list to tell the user what has changed
 when a change notification or conflict is issued?  My app burns a lot of
 resources because of this problem.

iterative map reduce

[Dominic Tarr]

hi,

I need a iterative map reduce, prefurably, in couchdb.

I want to be able to do a map reduce to generate unique items, then another
map reduce to generate stats about those unique items.

from what I know about couchdb at the moment, it seems like id have to do
the first step at the document creation step, which is just ugly.

has anyone else dealt with this problem?

Re: Document design: Logging accesses to an item by a user

[Dominic Tarr]

this might be easiest to achieve with a an update handler.

you can create all kinds of atomic transactional things like that, on a
single document though.

the user requests the update handler, which updates the count for that
user, and returns the song info, all in one request.

On Sun, Nov 13, 2011 at 3:25 PM, Manokaran K manoka...@gmail.com wrote:

 You can put it in the user doc or have a third doc. You can't have it on
 the song doc, because when many users are writing to it, it will cause rev
 mismatch errors.

 regds,
 mano

Re: unique constraint in couchdb

[Dominic Tarr]

so, a user may have a multiple timestamps, but they must all be unique?

what do the timestamps represent? is there some other feature that could be
in the key?

or, could you use a update handler and put all timestamps per user in the
same document?

On Mon, Oct 24, 2011 at 7:48 AM, Michael Zedeler mich...@zedeler.dk wrote:

 On 2011-10-20 15:19, Alexander Uvarov wrote:

 is there something on the roadmap for couchdb that would solve our unique
 key problem?

 There is no uniqueness in CouchDB and I think will never be.

 PS: I think that will be cool to have a feature to declare such
 constraints. And mark documents (if constraint void) just like them
 marked when conflict occurs (doc._conflict). Something like
 doc._constraints[unique_**timestamp]. Then it will be possible to
 create a view and retrieve a list of documents with constraints
 violation.

 But isn't this possible to do just by writing a view with a suitable reduce
 function?

 --
 Michael Zedeler
 70 25 19 99 | LinkedIn 
 http://dk.linkedin.com/in/**mzedelerhttp://dk.linkedin.com/in/mzedeler
 | Twitter 
 http://twitter.com/#%21/**mzedelerhttp://twitter.com/#%21/mzedeler
 | Github https://github.com/mzedeler

scalabitily of the _changes feed

[Dominic Tarr]

is it okay to register a huge number of listeners to the changes feed?

I notice that you can use a filter function, which gets passed the request
parameters,
does this mean that the filter function will be called for each doc, for
each listener on the changes feed?
or are the results of each filter call cached so that it is possible to
efficiently register many listeners using the same filter function?

cheers, Dominic

versioning views

[Dominic Tarr]

hi,

last time I did some couch db development,
I ran into some difficulties managing view versions.

so what I was thinking was, that this time i'll write a tool to manage their
versions.

basically, i'll save all the views with an id like: _design/whatever_v1.3.5

then, in my server code (which will be nodejs)

view ('whatever', '~1.2', 'all', function (err, data) {
...
})

when I start my server, it will request all views from the server,
decide which which is the latest view that matches a stated range, and use
that.

does this sound like a good idea, or am I mad?

see also:

http://semver.org

https://github.com/isaacs/node-semver

cheers, Dominic

Re: versioning views

[Dominic Tarr]

I think this will mainly be an advantage during development.

On Sat, Sep 24, 2011 at 2:34 AM, Mark Hahn m...@boutiquing.com wrote:

 The views for my app have to match the version of my app so I maintain them
 the same as I do any other source code.  Whenever I release I have a
 utility
 to push them to the db.

 On Fri, Sep 23, 2011 at 7:34 AM, Dominic Tarr dominic.t...@gmail.com
 wrote:

  hi,
 
  last time I did some couch db development,
  I ran into some difficulties managing view versions.
 
  so what I was thinking was, that this time i'll write a tool to manage
  their
  versions.
 
  basically, i'll save all the views with an id like:
 _design/whatever_v1.3.5
 
  then, in my server code (which will be nodejs)
 
  view ('whatever', '~1.2', 'all', function (err, data) {
  ...
  })
 
  when I start my server, it will request all views from the server,
  decide which which is the latest view that matches a stated range, and
 use
  that.
 
  does this sound like a good idea, or am I mad?
 
  see also:
 
  http://semver.org
 
  https://github.com/isaacs/node-semver
 
  cheers, Dominic
 

Re: Cross/Joined Database Access?

[Dominic Tarr]

you would have to do multiple http requests, i think.

would it be possible to combine all of the dbs into one database and just
have a field = user1,user2 etc?

somethings like this seem weird, coming from relational databases, but thats
couchdb.

On Tue, Mar 22, 2011 at 9:02 PM, Marc Schwering mschwer...@weluse.dewrote:

 Hi there.

 Is it possible // How do i Search on multiple Databases?

 Given one Layout/Schema of Entities:
 Postings = id; title; content; timestamp

 This Schema exists due multiple instances of one Application (for different
 Users) on n Databases. Like: db1,db2,db3

 I want to do two fetches:

 a) all Posts since time X until Y sorted by timestamp. from db1  db3.
 b) all Posts since time X until Y sorted by timestamp. from db2  db3.

 Is that possible? If so, how?

 Thanks in advance

 Marc

  --
 M.Schwering

Re: multiple database or many doc types in one database?

[Dominic Tarr]

right.

so what is needed is a script to migrate a type to (or from) it's own
database. and replicate the views.

hmm... i think you can do a one time replication with a filter...

then you'd just have to change the code which requests those views... or use
a proxy.




On Mon, Mar 21, 2011 at 10:37 PM, Paul Hirst paul.hi...@sophos.com wrote:

 On Thu, 2011-03-17 at 14:48 +, Kevin R. Coombes wrote:
 
  On 3/17/2011 3:46 AM, Paul Hirst wrote:
   On Thu, 2011-03-17 at 04:51 +, Dominic Tarr wrote:
   is it a good idea to
  
  1. separate documents for different types of thing into different
   databases?
  
   or is it better to
  
  2. just have a type='whatever' and check for that in your views?
   I am using option 2. I prepend type_ to my ids to avoid keys clashing
   and also have a type field in the body which I use in my views.
  
   However I'm really regretting doing it this way as I only have two
   documents types. I have 0.25 million of type A and 28 million of type
 B.
   I often want to change the views regarding type A documents and have to
   wait about 12 hours for it to rebuild. If I had separate databases this
   would be far less of a problem as the type A views would rebuild in a
   few minutes.
  What happens if you simply seprate the views into two different design
  documents?  If the views for type A were in one design document, then
  you would at least avoid rebuilding the views that work with type B
  (which at least suggest that the only time hit would come from having to
  execute if (type == 'A') emit(stuff) on the type B documents.

 Actually it's already separate design documents. But the cost of loading
 ~27.75 million documents from the disk, json encoding and sending to the
 view server is still very significant, even if the view server only does
 a simple 'if' test and doesn't return anything.

 This led me to think it would be great if documents could be tagged
 somehow and then views could choose which documents to include based on
 metadata tags. Then I figured this is almost exactly what I would get
 using different databases.



 Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP,
 United Kingdom.
 Company Reg No 2096520. VAT Reg No GB 991 2418 08.

Re: Some CouchDB internals questions?

[Dominic Tarr]

do you know about the include_doc option on view queries?

if you emit(key, doc) in the view, then the whole doc will be saved in the
view cache,
but if you query a view with the include_doc=true then the view results will
include the whole doc.
this is fetched from the db and not stored in the view.

this will slow reads, but use less space.
which option is best depends on your application.

it's mentioned here:

http://wiki.apache.org/couchdb/HTTP_view_API

On Thu, Mar 17, 2011 at 5:53 AM, Zdravko Gligic zgli...@gmail.com wrote:

 WOW !

 So, how long might it take for this not only to become part of CouchDB
 core but then also to get implemented by all of the ohter CouchDB
 dialects such as CouchBase and BigCouch ,etc ?

 And as dumb as it might sound ;) why was this not done (: the right
 way :) from the very beginning ;?)

 On Wed, Mar 16, 2011 at 10:02 AM, Filipe David Manana
 fdman...@apache.org wrote:
  Zdravko,
 
  Yesterday a performance related ticket was created:
 
  https://issues.apache.org/jira/browse/COUCHDB-1092
 
  Apart from the performance improvements, it also reduces very
  significantly the database sizes (from 2 times less to about 10 times
  less). So you might be interested to follow/read.
 
  On Tue, Mar 15, 2011 at 7:32 PM, Paul Davis paul.joseph.da...@gmail.com
 wrote:
  On Tue, Mar 15, 2011 at 2:53 PM, Zdravko Gligic zgli...@gmail.com
 wrote:
  Have you compacted your db and views?
 
  Yes
 
  There's unfortunately no direct way to calculate a upper threshold, it
  really depends on your method for inserting as well as how often you
  compact.
 
  Once both (docs and view) are compacted, is the resulting size at all
  dependent on how the docs and/or views were created in the first place
  (one at a time or in bulk or whatever) ?
 
 
  I think to get the absolute minimum post-compaction size you need to
  compact twice. I haven't done lots of extensive testing on this, but
  last I recall the basic logic was the first time can end up writing
  docs in a somewhat randomish ordering depending on how they were
  inserted.
 
  This is due to the tail append storage which will orphan data
  in the file as it writes new records to the various internal data
  structures.
 
  My 1,500 docs are taking up almost 15 meg (roughly 1/4-1k docs with 2
  views + 1 view with doc re-emit) and I believe were around 50meg
  before compactions.
 
 
  More importantly, what was the datasize post-compaction though? If
  your main db is 15Meg, and you have a view that re-emits the doc, I'd
  expect you to have a total size of at least 30Meg. Depending on what
  you're emitting in the other two views getting closer to that 50 isn't
  hugely out of the question.
 
 
 
 
  --
  Filipe David Manana,
  fdman...@gmail.com, fdman...@apache.org
 
  Reasonable men adapt themselves to the world.
   Unreasonable men adapt the world to themselves.
   That's why all progress depends on unreasonable men.
 

multiple database or many doc types in one database?

[Dominic Tarr]

is it a good idea to

  1. separate documents for different types of thing into different
databases?

or is it better to

  2. just have a type='whatever' and check for that in your views?

I'm starting to think that 2 is enough seperation, and 2 is silly.

are there any reason to use 2, except maybe to stop keys colliding? (which
is easy enough to do in other ways...)

cheers. Dominic