This project is read-only.

Very curious (different) memory issue

Feb 19, 2014 at 10:40 AM
Edited Feb 21, 2014 at 11:51 PM
Hello All!

So, thanks to the help of the community - I solved my last problem which was due to disposing of a handle which led to the gc() not being comprehensive.

All sorted!

Now I went to fix another issue (just one me picking up the wrong field as I record my results), and I ran into a very very strange one. Strange enough I think it might be a bug - but if not (or so) any workarounds anyone can offer me are muchly appreciated.

For more comprehensive code of mine you can see the last thread I wrote, but I don't think that's relevant. Here is a single difference which causes me to either run out of memory (leak), or otherwise complete successfully.

The code with the memory leak is -

DynamicVector testvar5 = rEngine.GetSymbol("mNames").AsVector();
NumericMatrix testvar14t = rEngine.GetSymbol("mCoef").AsNumericMatrix();

for (int i = 0; i < testvar14t.RowCount; i++)
{
__  //got to 5th x on the below
    //string tempIvN = "(Intercept)";

    //just up to 1st on this sucker
    string tempIvN = testvar14t.RowNames[i];__

    Single coeff = (Single)testvar14t[i, 0];
    Single pvalue = (Single)testvar14t[i, 3];

    if (Single.IsInfinity(coeff)) coeff = 0;
    if (Single.IsNaN(coeff)) coeff = 0;

    if (Math.Abs(coeff) < 1.0E-30) coeff = 0;
    if (Math.Abs(coeff) > 1.0E30) coeff = 0;

    if (Single.IsInfinity(pvalue)) pvalue = 1;
    if (Single.IsNaN(pvalue)) pvalue = 1;
    if (Math.Abs(pvalue) < 1.0E-30) pvalue = 0;
                                    
    DataRow oprNewRow = regnTable.NewRow();
    oprNewRow["SEC_ID"] = sid;
    oprNewRow["DAT_ID"] = c;
    oprNewRow["LINDEX_ID"] = x;
    oprNewRow["IVR_INDEX_ID"] = indVarCodes[tempIvN];
    oprNewRow["IVR_INDEX_COEFF"] = coeff;
    oprNewRow["IVR_INDEX_PVALUE"] = pvalue;
    regnTable.Rows.Add(oprNewRow);

}

testvar5.Dispose();
testvar5 = null;

testvar14t.Dispose();
testvar14t = null;
Switching around the comments, where the code works :
NumericMatrix testvar14t = rEngine.GetSymbol("mCoef").AsNumericMatrix();

for (int i = 0; i < testvar14t.RowCount; i++)
{
__  //got to 5th x on the below
    string tempIvN = "(Intercept)";

    //just up to 1st on this sucker
    //string tempIvN = testvar14t.RowNames[i].ToString();__

    Single coeff = (Single)testvar14t[i, 0];
    Single pvalue = (Single)testvar14t[i, 3];

    if (Single.IsInfinity(coeff)) coeff = 0;
    if (Single.IsNaN(coeff)) coeff = 0;

    if (Math.Abs(coeff) < 1.0E-30) coeff = 0;
    if (Math.Abs(coeff) > 1.0E30) coeff = 0;

    if (Single.IsInfinity(pvalue)) pvalue = 1;
    if (Single.IsNaN(pvalue)) pvalue = 1;
    if (Math.Abs(pvalue) < 1.0E-30) pvalue = 0;

    DataRow oprNewRow = regnTable.NewRow();
    oprNewRow["SEC_ID"] = sid;
    oprNewRow["DAT_ID"] = c;
    oprNewRow["LINDEX_ID"] = x;
    oprNewRow["IVR_INDEX_ID"] = indVarCodes[tempIvN];
    oprNewRow["IVR_INDEX_COEFF"] = coeff;
    oprNewRow["IVR_INDEX_PVALUE"] = pvalue;
    regnTable.Rows.Add(oprNewRow);

}

testvar5.Dispose();
testvar5 = null;

testvar14t.Dispose();
testvar14t = null;
It's as if looking at " string tempIvN = testvar14t.RowNames[i].ToString(); " is killing me.


Why could this be, and how can I work around it ?


THANKS


***edit: removed flagrant use of f word
Feb 21, 2014 at 8:35 AM
Indeed there seems to be something fishy going on with character vectors to .NET string arrays.

Opened https://rdotnet.codeplex.com/workitem/84

Thanks for the report.
Feb 21, 2014 at 11:55 PM
Experimenting now using "row.names" in R [r.Engine.Execute] to load into a separate object, then bring back into a character vector [rEngine.GetSymbol] - rather than use the property of the NumericMatrix object.

Will report back.
Feb 22, 2014 at 12:12 AM
Note that I did get some memory leaks too using rownames in R and As.Character().ToArray(), though possibly less pronounced than using the RowNames property.
Feb 22, 2014 at 3:02 AM
Bugger!

Is there any version of a .Execute("gc()") or .ForceGarbageCollection() which will release the 'leaked' memory? I know the R Engine cannot be closed and restarted - but is there any sort of facsimile as far as memory is concerned?
Feb 22, 2014 at 8:56 AM
I do not see any workaround, and so far the code I looked at looks clean of any obvious issue. This will take some time to figure out, probably involving some trial and error. Well, in any case rolling up sleeves and coffee is required.
Feb 22, 2014 at 11:55 PM
No worries. What I am doing now is loading a lookup matrix into R so I can do the lookup there via Execute, then just bring back an integer vector.
Nonetheless: still had problems.

All becoming too mysterious -------- Rather than my crappy method of implement an idea, and see if I run out of memory 12 hours later, I'm writing some things to look at memory usage as I go, hopefully I can track this and find which configurations have the leak and which don't...
Feb 23, 2014 at 4:55 AM
Hi,

I think I understand what is going on. There is no memory leak as such in R.NET, however reading or writing large character vectors can blow memory footprint indeed. It seems that the disposal of transient SafeHandle objects (InternalString) by the .NET interop services takes much longer than their creation. So, if one keeps on triggering the creation of such objects faster than their disposal, one ends up with memory increase.

I'll see what I can do about it. I am hopeful that the R to .NET conversion can be worked around; the opposite may be trickier.
Feb 23, 2014 at 6:27 AM
Hi,

I have commited changes that seem to alleviate if not remove issues.
https://rdotnet.codeplex.com/SourceControl/changeset/8ec79d205661

While basic unit tests are in place, this needs a good real world testing to confirm the fixes.
Mar 8, 2014 at 4:57 AM
terrific! will examine.

I hadn't written back because I had solved the issue another way (poor of me I know).

Looks like I had (probably) two issues which were causing the memory leak.

1) Bringing back the character vector via .RowNames() was screwing me --- so what I did was to create a lookup table in R with an integer representing the name of each field I was using in my model. This way, RdotNet only needed to bring back a integer vector for me to tell what field I was looking at. Neat!

2) For some reason the testvar5 = null; (and instructions like this, setting things to null) was causing a memory leak. Perhaps something like the decoupling of handles issue I had previously, but nonetheless, when the variables were set to null I had the leak, when I just disposed I did not!

Thought I would report back how I got things to work!
I'll experiment with the new code also.