Strange Out of memory error.

[Luke]

Hi,

Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?
I tried making the array global, but that didn't make any difference.
--
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#COMPILER PBCC 4.04
#COMPILE EXE
#DIM ALL
$INCLUDE "Win32API.inc"
'----------------------------------------------------------------------------------
FUNCTION PBMAIN () AS LONG
LOCAL Elements AS DWORD,ms AS MEMORYSTATUS
TRY
ms.dwLength = SIZEOF(MEMORYSTATUS)
GlobalMemoryStatus ms
Elements = ms.dwAvailPhys/%MAX_PATH
DIM myArray(1 TO Elements) AS LOCAL ASCIIZ * %MAX_PATH
CATCH
? "Error: " + FORMAT$(ERR)
END TRY
? "Availablememory: " + FORMAT$(ms.dwAvailPhys,"#,") + " bytes"
? "Elements: " + FORMAT$(Elements,"#,")
? "Press a key..."
WAITKEY$
END FUNCTION

[Michael Mattias]

Strangely enough, the error message is exactly correct: you are trying to
allocate too much memory. Your available physical must now be such that the
total requirements for your program when you try to allocate all of that
'available physical' exceed your 2 Gb user limit per process. That limit
includes any space required for other things in your program.. eg, your
code, your stack, progam overhead, etc.

To execute that REDIM you need #elements * %MAX_PATH bytes VIRTUAL memory
available.

Your max remaining space in your process is the dwAvailVirtual member.But I
would not bet a lot on the ability actually allocate that much. I could
never get more than 1.5 Gb (virtual) on my machine.

Just out of curiosity, why do you even bother trying to allocate that many
elements of that size? It's gonna get swapped out anyway.. and you can get
the benefit of the page files with memory-mapped file object without the
cost of not having any memory left for other things.

Not to mention, "physical" memory doesn't really mean much under Windows,
where everything is virtual.
--
Michael C. Mattias
Tal Systems Inc.
Racine WI

Post by Luke
Hi,
Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?
I tried making the array global, but that didn't make any difference.
--
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#COMPILER PBCC 4.04
#COMPILE EXE
#DIM ALL
$INCLUDE "Win32API.inc"
'----------------------------------------------------------------------------------
FUNCTION PBMAIN () AS LONG
LOCAL Elements AS DWORD,ms AS MEMORYSTATUS
TRY
ms.dwLength = SIZEOF(MEMORYSTATUS)
GlobalMemoryStatus ms
Elements = ms.dwAvailPhys/%MAX_PATH
DIM myArray(1 TO Elements) AS LOCAL ASCIIZ * %MAX_PATH
CATCH
? "Error: " + FORMAT$(ERR)
END TRY
? "Availablememory: " + FORMAT$(ms.dwAvailPhys,"#,") + " bytes"
? "Elements: " + FORMAT$(Elements,"#,")
? "Press a key..."
WAITKEY$
END FUNCTION

[Luke]

Post by Michael Mattias
Strangely enough, the error message is exactly correct: you are trying to
allocate too much memory. Your available physical must now be such that
the total requirements for your program when you try to allocate all of
that 'available physical' exceed your 2 Gb user limit per process.

I'm not sure I understand you correctly. The help file says, under Features
of PBCC, "Use of all available memory(up to 2GB) for arrays and strings"
In my case I have 1.3GB memory available.And there was no problem with 1GB
memory installed.

Post by Michael Mattias
That limit includes any space required for other things in your program..
eg, your code, your stack, progam overhead, etc.
To execute that REDIM you need #elements * %MAX_PATH bytes VIRTUAL memory
available.
Your max remaining space in your process is the dwAvailVirtual member.But
I would not bet a lot on the ability actually allocate that much. I could
never get more than 1.5 Gb (virtual) on my machine.
Just out of curiosity, why do you even bother trying to allocate that many
elements of that size? It's gonna get swapped out anyway.. and you can get
the benefit of the page files with memory-mapped file object without the
cost of not having any memory left for other things.

The program don't know the number of elements in advance, so it just pick a
number of elements within available memory, and later, when number of
elements is known, the array is being redimed.

Maybe I could just write it all to a file instead of using an array.
--

Post by Michael Mattias
Not to mention, "physical" memory doesn't really mean much under Windows,
where everything is virtual.
--
Michael C. Mattias
Tal Systems Inc.
Racine WI

Post by Luke
Hi,
Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?
I tried making the array global, but that didn't make any difference.
--
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#COMPILER PBCC 4.04
#COMPILE EXE
#DIM ALL
$INCLUDE "Win32API.inc"
'----------------------------------------------------------------------------------
FUNCTION PBMAIN () AS LONG
LOCAL Elements AS DWORD,ms AS MEMORYSTATUS
TRY
ms.dwLength = SIZEOF(MEMORYSTATUS)
GlobalMemoryStatus ms
Elements = ms.dwAvailPhys/%MAX_PATH
DIM myArray(1 TO Elements) AS LOCAL ASCIIZ * %MAX_PATH
CATCH
? "Error: " + FORMAT$(ERR)
END TRY
? "Availablememory: " + FORMAT$(ms.dwAvailPhys,"#,") + " bytes"
? "Elements: " + FORMAT$(Elements,"#,")
? "Press a key..."
WAITKEY$
END FUNCTION

[Michael Mattias]

Post by Luke

Post by Michael Mattias
Strangely enough, the error message is exactly correct: you are trying to
allocate too much memory. Your available physical must now be such that
the total requirements for your program when you try to allocate all of
that 'available physical' exceed your 2 Gb user limit per process.

I'm not sure I understand you correctly. The help file says, under
Features of PBCC, "Use of all available memory(up to 2GB) for arrays and
strings"
In my case I have 1.3GB memory available.And there was no problem with 1GB
memory installed.

While it appears from other replies your problem may be related to SP3, let
me offer this.....

2 Gb is the ideal, the theoretical. I ran tests here using PB (forgot if I
used CC or WIN) and could never get more than 1.5 Gb allocated to my
process. It may have someting to do with allowed page file size or some
other system setting (Win/XP Pro SP2 all updates). I thought of one more
thing to try but I haven't got to that yet.

Post by Luke
The program don't know the number of elements in advance, so it just pick
a number of elements within available memory, and later, when number of
elements is known, the array is being redimed.
Maybe I could just write it all to a file instead of using an array.

Well, no reason you can't use both... as long as you process in "some
reasonable size" chunks.

Your demo code doesn't really describe the application, but since you say
you don't know the number of elements in advance I assume it is something
like, "select all records meeting some condition from a file or database and
do something with them." Maybe if you describe the application a little
better (in English, not in code) someone can come up with some processing
options for you which will be both fast enough and efficient in its use of
system resources.

The choice to use "avaiable physical" memory is kind of a curious
choice...as I mentioned in my prior post, the physical memory available is
essentiallly a meaningless number, except in some very, very specialized
applications which take over the whole machine and do things at a very very
deep level... things you would not do in a normal business application.
(Gamers might do some of these things; so might applications which control
hardware).

I think posting a one or two paragraph description of your application with
some "typical" volume info would get you some pretty good ideas of how to
get your job done without running into memory-allocation issues.

[Peter Manders]

Post by Luke

Post by Michael Mattias
Strangely enough, the error message is exactly correct: you are trying to
allocate too much memory. Your available physical must now be such that
the total requirements for your program when you try to allocate all of
that 'available physical' exceed your 2 Gb user limit per process.

I'm not sure I understand you correctly. The help file says, under Features
of PBCC, "Use of all available memory(up to 2GB) for arrays and strings"
In my case I have 1.3GB memory available.And there was no problem with 1GB
memory installed.

While I agree with Michael's answer to you, I have a suggestion for a
possible generic approach. Why don't you use a binary kind of scan to
find out how much memory you can allocate? Something like:

Can I get 2Gb?
If no, can I get 1Gb?
If yes, can I get 1.5Gb?
If yes, can I get 1.75Gb?
if no, can I get 1.625Gb?

Etcetera until the delta amount is too small to bother.

Also, maybe you do have something like 1.2Gb free memory, but it could
be split up so the largest amount you can actually allocate may be
just half of it (depending on how well Windows manages memory
allocation).

[Gerard Bok]

Post by Luke
Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?

For what it is worth:
I compiled your snippet with PBCC 4 and it runs without problems
on both Windows 98 (512 MB: 229,224,448 bytes, 881,632 elements)
and Windows XP SP2 (2 GB: 1,437,659,136 bytes, 5,529,458
elements).

(As my XP laptop uses an AMD CPU, I am not to keen to install SP3
yet :-)

[Luke]

Post by Gerard Bok

Post by Luke
Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?

I compiled your snippet with PBCC 4 and it runs without problems
on both Windows 98 (512 MB: 229,224,448 bytes, 881,632 elements)
and Windows XP SP2 (2 GB: 1,437,659,136 bytes, 5,529,458
elements).
(As my XP laptop uses an AMD CPU, I am not to keen to install SP3
yet :-)
--
Kind regards,
Gerard Bok

Hmmm...

This points toward SP3 as being the problem, maybe.
As I said before, it runs fine on WIN XP SP2 with 1 GB memory.
--

[Wolfgang Enzinger]

Post by Luke
This points toward SP3 as being the problem, maybe.
As I said before, it runs fine on WIN XP SP2 with 1 GB memory.

I tend to think that it's a hardware problem (memory?) on your
machine, cause the app runs fine here on XP SP3 with 2 GB memory:

Availablememory: 1,109,266,432 bytes
Elements: 4,266,409
Press a key...

Wolfgang

[Luke]

Post by Wolfgang Enzinger

Post by Luke
This points toward SP3 as being the problem, maybe.
As I said before, it runs fine on WIN XP SP2 with 1 GB memory.

I tend to think that it's a hardware problem (memory?) on your
Availablememory: 1,109,266,432 bytes
Elements: 4,266,409
Press a key...
Wolfgang

Maybe it is so, but all of Dell's memory diagnostic tests passed with no
errors reported.
--

[Michael Mattias]

Post by Luke

Post by Wolfgang Enzinger

Post by Luke
This points toward SP3 as being the problem, maybe.
As I said before, it runs fine on WIN XP SP2 with 1 GB memory.

I tend to think that it's a hardware problem (memory?) on your
Availablememory: 1,109,266,432 bytes
Elements: 4,266,409
Press a key...
Wolfgang

Maybe it is so, but all of Dell's memory diagnostic tests passed with no
errors reported.

As I said earlier in thread.... hardware and operating system differences
may be the proximate cause of this particular error, but the design itself
is curious, and easy to change to avoid the problem altogether regardless of
hardware or O/S.

MCM

[Gerard Bok]

Post by Luke

Post by Gerard Bok

Post by Luke
Below is shown a simplified version of a construct I have been used for
years without any problem, but now it produce an error 7(Out of memory).
What's new lately is that I have upgraded my memory from 1 to 2GB, and
upgraded Windows XP from SP2 to SP3.
Have anyone experienced memory problem with WinXP SP3?

I compiled your snippet with PBCC 4 and it runs without problems
on both Windows 98 (512 MB: 229,224,448 bytes, 881,632 elements)
and Windows XP SP2 (2 GB: 1,437,659,136 bytes, 5,529,458
elements).
(As my XP laptop uses an AMD CPU, I am not to keen to install SP3
yet :-)

Hmmm...
This points toward SP3 as being the problem, maybe.
As I said before, it runs fine on WIN XP SP2 with 1 GB memory.

This may be no longer relevant :-)
But yet: I installed SP3 and came around a copy of the program.

Runs flawless. AMD Turion 64, single core, XP Home SP3, 2 GB
RAM. (But onboard video, which takes a heap out of those 2 GB.)