Just a short follow up to the list. It turns out that the strange behavior I was experiencing had nothing to do with cout and cerr, but ostreams in general. For some reason, at a minimum, the operators: ostream& operator<< (long& val ); ostream& operator<< (double& val ); were not functioning properly and were causing my problem. To get around this, I simply had to convert types like these to strings before passing them into the ostreams with << operators. I don't know if this was due to a mistake in which libraries I linked to or which headers are being read when I make my project. However, it seems that I have ruled out the possibility of this being unique to 64-bit longs as the doubles are standard types on any machine. Anyway, I appreciate everyone's help, and if anyone has trouble with this type of thing in the future, a simple couple of functions I used to convert the long or double values to strings were: string getStringFromDouble(double d) { char*doubleValue = (char*)calloc(1000,sizeof(char)); sprintf(doubleValue,"%f",d); string returnValue(doubleValue); free(doubleValue); return(returnValue); } string getStringFromLong(long i) { char*intValue = (char*)calloc(1000,sizeof(char)); sprintf(intValue,"%ld",i); string returnValue(intValue); free(intValue); return(returnValue); } I'm not sure why this was necessary, but it solves my issue. Thanks again for all the help--I learned a lot about streams and stream buffers in the process. Thanks, Sean
On 5/8/11 12:56 AM, "Sean Robert McGuffee" <sean.mcguf...@gmail.com> wrote: > Thanks, Simon, > That link may prove to be very useful. I think what I can do is make a > similar version of a buffer to one in that example and then have it > conditionally write to Rprintf or cout and/or REprintf or cerr, > depending on whether I'm running a c++ program from the command line > or launching a function from R. Then I can just replace all of my cout > and cerr terms with this new buffer or two in global space. Before I > call those functions in R, I'll set them to write to R before I launch > any code that uses it. This should allow me to leave all my code > exactly how it is with a minor replacement of one or two terms for > another. So, if the errors in my performance are only caused by > writing to cout and/or cerr from R, then they can be completely > avoided by all of those substitutions writing to Rprintf and REprintf. > This seems like it is going to work exactly as I'm hoping. > Thanks again, > Sean > > On Sat, May 7, 2011 at 6:13 PM, Simon Urbanek > <simon.urba...@r-project.org> wrote: >> Sean, >> >> just to clarify - this is not about something "not working" or any leaks. >> cout/cerr do exactly what they are supposed to, it's just in most cases not >> what you want. The main reason is that R does not necessarily use >> stdin/stdout so cout/cerr may have nothing to do with the R console output. >> The effects of using cout/cerr vary by the actual setup in which you run R as >> I was explaining earlier. In the most complex setup (default for interactive >> R on OS X, actually) you have: >> >> 1) stdin/stdout/stderr - those are entirely independent of the R console >> 2) ReadConsole/WriteConsole - this is how R interacts with the console >> 3) C++ cin/cout/cerr - by default a buffered link to stdin/out/err >> >> So as you see the reason for cout being strongly discouraged is simply >> because it has no relationship with R and produces no output in R itself. >> What makes things seemingly puzzling is that some UI implementations in R use >> stdin/out/err in Read/WriteConsole, so suddenly you are dealing with three >> independent streams and buffers (e.g., cout, WriteCondole and stdout all >> eventually ending in the same place but at different times). If your final >> output is a tty then they all get synchronized at the end into the same tty, >> but since each of them has a buffer, the result is random at the best. >> >> Due to the interactions, you must be careful not to mess things up with any >> additional low-level stream manipulations. For example, the R.app GUI doesn't >> use stdout/err for console at all, since it is driven by text views instead, >> so it uses exclusively Read/WriteConsole (and so does the Windows GUI). >> However, it also monitors stdout/err for external output (e.g. from system()) >> and uses two separate pipes to merge that output into the console with >> different colors. So if you were to start to mess with the descriptors for >> stdout/err you could break that setup. >> >> That said, if you want a solution for C++ you should simply write a stream or >> buffer that uses Rprintf/REprintf for output. I was assuming that others have >> done that before, but Dirk's response suggests that it is not as common. If >> you choose to do it, you may find this link helpful: >> http://groups.google.com/group/comp.lang.c++/msg/1d941c0f26ea0d81 >> >> Cheers, >> Simon >> >> >> On May 7, 2011, at 12:42 AM, Sean Robert McGuffee wrote: >> >>> This is a good idea as to converting to stringstream's because it >>> could let me remove cout and cerr from the equation. At the moment, I >>> get that feeling from the experts that having cout and cerr is causing >>> a bug by not mixing with R. I have other cases where algorithms that >>> avoid cout and cerr aren't working, but as far as my experience with >>> bugs in c code goes, a memory issue in one place can make everything >>> go haywire, and an random cout or cerr somewhere that I don't catch >>> could maybe be the source of the issue. I also like this idea of >>> piping and then calling Rprintf. Along the same lines, I could create >>> a string and then either spit to cout or cerr or spit to Rprintf >>> depending on some sort of setting. One of the key reasons I have been >>> hanging on to cout and cerr is that I want to keep all of my c++ code >>> identical to the status quo as far as the command line interface is >>> concerned. On the other hand, maybe I can just globally substitute >>> cout and cerr to random file streams and avoid the conflict with >>> stdout and stderr. I take it that it isn't ostreams per say causing >>> the problem but that it's something specific about cout and cerr not >>> mixing with R. Then maybe I can pass their buffers to cout, cerr, or >>> Rprintf, depending on preferences later. What I would really like to >>> do maybe be beyond my grasp, but that would be to integrate ostreams >>> into R. I am toying with an idea of mimicking cout and cerr with some >>> new classes that are almost verbatim copies of cout and cerr with >>> substituted names. Then maybe inside their internals I can more easily >>> create conditional access to Rprintf calls. What I like about this >>> strategy is that I could do a global substitution of cout and cerr >>> with new objects of more useful functionality, and that way I might >>> not have to re-write all my c++ code which constantly calls on these >>> standard streams. Plus, if I get something generic to work, maybe I >>> can pass it on to Dirk so others won't have to deal with this stuff in >>> the future. I find this fascinating that the output may be tty. I see >>> now that the device on my Mac that /dev/stdout points to is a tty. I >>> have never encountered tty before, but I have a vague memory of a term >>> like that back in the day of dial up! Oh the fun things I'm learning. >>> Just to clarify though, is the only way my code "mixes IO systems >>> between R and C++" by way of cout and cerr happening to be >>> synchronized with stderr and stdout by default? Or is there something >>> inherent in ostreams in general that doesn't mesh with R? I guess I >>> find it a little bit surprising that calling >>> ios_base::sync_with_stdio(false) and redirecting those streams to >>> other files doesn't dodge that bullet. I know that doesn't break the >>> ostreams themselves because it works fine from the command line >>> interface. I guess I'll have to delve more into how c++ works behind >>> the scenes to figure that part out. I think Dirk has a really good >>> suggestion to comment out all of my cout and cerr calls and see if >>> still get a bug. Obviously that wouldn't be any permanent fix for >>> future users who want to link code with cout and cerr, but it would >>> help me identify if this is the only source of error. It might take me >>> till past the weekend to get to the bottom of some of these things, >>> but I'll post what I find out. >>> Thanks again--I appreciate everyone's useful suggestions. >>> Sean >>> >>> On Fri, May 6, 2011 at 10:44 PM, Dirk Eddelbuettel <e...@debian.org> wrote: >>>> >>>> On 6 May 2011 at 22:09, Simon Urbanek wrote: >>>> | Sean, >>>> | >>>> | the path form the console is not under your control and it depends >>>> heavily on the internal settings in R (is console enabled, is R >>>> interactive, is the output a tty or a file ...) - that's why you have to >>>> use Rprint/REprintf - that's the only path that is guaranteed to work. >>>> Anything else is settings-specific and not guaranteed to work (it may >>>> appear to work in limited settings, but will not work in others). Note >>>> that, for example, WriteConsole is only used if R is not fed via I/O pipes >>>> - that may be what baffled you in that case. >>>> | >>>> | So AFAICS the only C++ solution (other than using Rprintf/REprintf >>>> directly) is to create streams to pipes whose ends will call >>>> Rprintf/REprintf with the content. Anything else is not general. You may >>>> want to ask Rcpp people (Dirk is CC'd anyway ;)) - I'd imagine they must >>>> have thought of it as it seems very natural to request in a C++ interface >>>> ... (but I don't use Rcpp so I don't know). >>>> >>>> Rcpp makes no claims about reimplementing IO, or providing new IO >>>> facilities. >>>> Rcpp is clearly bound by the R API upon which it builds; the R >>>> documentation >>>> (which I quoted in my earlier email in this thread) suggests to expect >>>> nasty >>>> things when one mixes IO systems between R and C++. >>>> >>>> And that seems to be what Sean is experiencing; I for one am not surprised. >>>> It was never clear from Seans's mails exactly how much time he has had for >>>> reading of the available documentation, his approach seems to me mostly >>>> driven by a vast and vague set of assumption on his part (eg the rout/rerr >>>> business he comes back to), as well as a lot energy and a willingness to >>>> just >>>> try this. Which is perfectly fine, but it can also fail. >>>> >>>> I'd suggest to start small with as little IO as possible. Maybe even >>>> comment >>>> all IO out as much as possible. Or redo it as strstream and feed that to >>>> Rprintf. >>>> >>>> Dirk >>>> >>>> >>>> | Cheers, >>>> | Simon >>>> | >>>> | >>>> | >>>> | On May 6, 2011, at 3:19 PM, Sean Robert McGuffee wrote: >>>> | >>>> | > Hi, >>>> | > >>>> | > Thanks for the comments so far. >>>> | > >>>> | > I've been going through the code extensively and seem to be having >>>> trouble >>>> | > reproducing what R is doing, so this confuses me. For example, when I >>>> write >>>> | > out the pointer values for ptr_R_WriteConsole type functions, I can't >>>> find a >>>> | > function that matches what is actually being called. >>>> | > >>>> | > REprintf("Rstd_WriteConsole=%ld, ptr_R_WriteConsole=%ld, >>>> | > R_WriteConsole=%ld\n", Rstd_WriteConsole, ptr_R_WriteConsole, >>>> | > R_WriteConsole); >>>> | > >>>> | > It might be something like RTcl_WriteConsole too. However, I can't seem >>>> to >>>> | > find the headers I would need to try to find it's pointer value. >>>> Anyway, >>>> | > these are often hidden/protected/static which is an aspect of code that >>>> I >>>> | > only understand in concept. The concept I understand about this type of >>>> | > stuff is that if you provide a better way of doing things, then maybe >>>> you >>>> | > hide some stuff that doesn't involve the better way. However, in this >>>> case, >>>> | > the hidden stuff is just making it difficult for me to debug and get to >>>> | > bottom of what is going on. Anyway, I get different results if I call >>>> on the >>>> | > functions that simply use stdout when trying to reproduce Rprintf >>>> | > functionality. I'm only doing that to try to get to the bottom of what >>>> R is >>>> | > doing with stdout and stderr. I also am having a similar issue with >>>> trying >>>> | > to understand what GNU GCC is doing with how it sync's cout and cerr >>>> with >>>> | > stdout and stderr. Maybe R is intercepting some of these components. >>>> Anyway, >>>> | > >>>> | > I was hoping to get around the >>>> | > "5.6 Interfacing C++ code >>>> | > ======================== [...] >>>> | > Using C++ iostreams, as in this example, is best avoided. There >>>> is >>>> | > no guarantee that the output will appear in the R console, and indeed >>>> it >>>> | > will not on the R for Windows console. Use R code or the C entry >>>> points >>>> | > (*note Printing::) for all I/O if at all possible." >>>> | > if at all possible. My reason is that I have 76386 lines of c++ code >>>> that >>>> | > I'm trying to interface with R, and it literally might be faster for me >>>> to >>>> | > update R rather than modify all of that code to suit R which seems to >>>> be >>>> | > lacking a c++ ostream interface. >>>> | > >>>> | > Ideally, I'd write rout and rerr versions of cout and cerr for R to >>>> become >>>> | > more compatible with c++. I have had some trouble figuring a lot of >>>> this >>>> | > hidden stuff out. Strategies I have considered are the following: >>>> | > 1) Write macros that conditionally replace cerr and cout with Rprintf >>>> | > somehow throughout all of my c++ code that I'm trying to interface with >>>> R. >>>> | > 2) Recreate the std::cerr and std::cout type of classes but maybe add >>>> some >>>> | > extra functionality for callbacks to when i/o events take place to >>>> somehow >>>> | > try to fix things. >>>> | > 3) Make a scratch version of R source code that has no hidden aspects >>>> to try >>>> | > to investigate what is going on a little better. >>>> | > >>>> | > Also, I should mention that I found an unanswered post from someone who >>>> had >>>> | > a similar problem when trying to use cerr and cout when accessing >>>> functions >>>> | > from dynamically linked libraries. I wonder, is it possible that R >>>> hasn't >>>> | > completely connected to these libraries? I mean, it's doing some sort >>>> of >>>> | > magic where it finds a pointer to a function, but maybe c++ has a lot >>>> going >>>> | > on in the background related to these ostream types that could be >>>> getting >>>> | > messed up. Maybe some aspects of the global environment might be >>>> different >>>> | > somehow? I directly link to my libraries via a c++ compiler when I get >>>> the >>>> | > identical functions to work from main in c++. >>>> | > >>>> | > One final thing I'm considering is that the R manual also mentions >>>> something >>>> | > about garbage collection and some circumstances under which it's >>>> important >>>> | > to explicitly request pointers to be maintained. Is it possible that R >>>> is >>>> | > eating global pointers used by std::ostream ? >>>> | > >>>> | > All suggestions are valued and if anyone has recommendations on >>>> strategy, >>>> | > please let me know. >>>> | > >>>> | > Thanks, >>>> | > Sean >>>> | > >>>> | > >>>> | > On 5/6/11 2:50 PM, "Davor Cubranic" <cubra...@stat.ubc.ca> wrote: >>>> | > >>>> | >> On 2011-05-06, at 11:41 AM, Dirk Eddelbuettel wrote: >>>> | >> >>>> | >>> | I¹m trying to call some of my c++ code from R and seem to be having >>>> an >>>> | >>> issue >>>> | >>> | with streams, although that¹s just one obvious sign of something >>>> different >>>> | >>> | in performance between calling the same function from main in c++ >>>> vs. >>>> | >>> | calling the same function from R. [...] >>>> | >>> In a nutshell, R 'owns' your console. That is part of the Faustian >>>> bargain. >>>> | >> >>>> | >> But isn't he redirecting cout to a file, so the C++ output is not to >>>> the >>>> | >> console? >>>> | >> >>>> | >> Davor >>>> | > >>>> | > >>>> | > On 5/6/11 2:41 PM, "Dirk Eddelbuettel" <e...@debian.org> wrote: >>>> | > >>>> | >> >>>> | >> On 6 May 2011 at 14:21, Sean Robert McGuffee wrote: >>>> | >> | Hi, >>>> | >> | >>>> | >> | Sorry, I just tried posting this but I had it in the wrong format of >>>> text, >>>> | >> | so this is a cleared format repost: >>>> | >> | >>>> | >> | I¹m trying to call some of my c++ code from R and seem to be having >>>> an issue >>>> | >> | with streams, although that¹s just one obvious sign of something >>>> different >>>> | >> | in performance between calling the same function from main in c++ >>>> vs. >>>> | >> | calling the same function from R. I¹m not getting any signs of >>>> errors from >>>> | >> | R, but the behavior is different somehow and in a way that breaks my >>>> | >> | algorithms. In both cases of launching a function from c++ or R, I >>>> redirect >>>> | >> | cout to a file using a streambuf. In particular, I can see progress >>>> stop >>>> | >> | when launching from R at a point when cout gets truncated when >>>> starting to >>>> | >> | write out iterator values in a std::vector<string>. Then, I find no >>>> way to >>>> | >> | write anything else to cout. When I write those values to Rprintf, >>>> they are >>>> | >> | there and they are as expected. I¹ve tried setting >>>> | >> | ios_base::sync_with_stdio(false) in case R is messing with stdout >>>> and stderr >>>> | >> | somehow, but that made no difference. I¹m not quite sure how R uses >>>> those C >>>> | >> | FILE* streams. Anyway, I am wondering if anyone has a hunch as to >>>> what might >>>> | >> | be going on. At this point I don¹t know if it might be an issue with >>>> R not >>>> | >> | completely handling c++ from a library or if it is an issue with >>>> cout in >>>> | >> | particular or if there is something else I¹m missing. Please let me >>>> know if >>>> | >> | you have any suggestions for me. >>>> | >> >>>> | >> Doug Bates sometimes remarks that, once all other alternatives are >>>> exhausted, >>>> | >> one could consider reading the manual. And indeed, 'Writing R >>>> Extensions' has >>>> | >> this to say: >>>> | >> >>>> | >> 5.6 Interfacing C++ code >>>> | >> ======================== >>>> | >> >>>> | >> [...] >>>> | >> >>>> | >> Using C++ iostreams, as in this example, is best avoided. There >>>> is >>>> | >> no guarantee that the output will appear in the R console, and >>>> indeed it >>>> | >> will not on the R for Windows console. Use R code or the C entry >>>> points >>>> | >> (*note Printing::) for all I/O if at all possible. >>>> | >> >>>> | >> In a nutshell, R 'owns' your console. That is part of the Faustian >>>> bargain. >>>> | >> >>>> | >> Dirk >>>> | >> >>>> | >> -- >>>> | >> Gauss once played himself in a zero-sum game and won $50. >>>> | >> -- #11 at http://www.gaussfacts.com >>>> | > >>>> | > ______________________________________________ >>>> | > R-devel@r-project.org mailing list >>>> | > https://stat.ethz.ch/mailman/listinfo/r-devel >>>> | > >>>> | > >>>> | >>>> >>>> -- >>>> Gauss once played himself in a zero-sum game and won $50. >>>> -- #11 at http://www.gaussfacts.com >>> >>> >> >> ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel