SubEtha Users <users@subethamail.org>
From Jon Stevens Jul 1, 2009 12:01 PM
Umm yea, I'd just turn off mail queue approval before doing the import and then turn it back on afterwards. =) jon On Wed, Jul 1, 2009 at 12:23 AM, Charles Williams <c.williams@planet-ic.de>wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jon, > > so. after the change I can now import a 90+ MB archive into subetha which > is great. the only problem > is that the archive is not imported directly into the archive in subetha. > instead it is put in the > mail queue for approval (list holds all new messages). which means there > are about 400 mails waiting > for approval and once approved they will be sent to all list members. > > not a very good way to import old archives. > > chuck > > > Charles Williams wrote: > > Jon, > > > > that's what I ended up doing yesterday. bumped it to 100 MB (exim is > limited to 50 MB anyway) and > > hopefully that will be enough. it did take forever to find the correct > constant to change though. > > why couldn't you have answered earlier? LOL > > > > anyway. am going to try and import an archive now and see if that works > now. I hope so. have been > > dealing with this migration for over a month now. has long since become > old. :( > > > > thanks again, > > chuck > > > > > > Jon Stevens wrote: > >> It is odd that you are hitting this limit with a 4 meg attachment, but I > can > >> easily see it with a larger (14/35meg) attachment. > > > >> The issue is buffering. Imagine if 100 messages come in at the same > exact > >> time and each has a 10 meg attachment. That is a minimum of 1gig of > memory > >> that the JVM needs to deal with. If you overflow the JVM heap size, then > you > >> are shit out of luck. (Make sure you increase the amount of memory > allocated > >> to the JVM... edit resin.conf and specify a much larger > >> <jvm-arg>-Xmx512m</jvm-arg>... I'd also specify an equally sized -Xms > >> variable as well.) > >> Agreed that this should be a more easily configured option. We can > improve > >> on that in the future. Have you tried increasing the limit? (just > recompile > >> subetha, it is pretty easy). Change the MAX_MESSAGE_BYTES variable in > the > >> InjectorBean to something much larger. Here is the link to how to setup > a > >> development environment. You can skip the eclipse part... just check the > >> source code out and use ant to compile a war file for you with an 'ant > war' > >> command. > >> http://code.google.com/p/subetha/wiki/DeveloperGuide > > > >> jon > > > > > >> On Mon, Jun 29, 2009 at 11:48 PM, Charles Williams > >> <c.williams@planet-ic.de>wrote: > > > >> well, just finished a google search regarding this and it seems to be > the > >> same error/bug that was > >> first reported back in 2007. which impacts all mail whether imported > into > >> the archives or newly sent > >> messages. > > > >> this is also the same problem that I reported a while back. and am > >> currently catching hell for here. > >> am wondering why a hardlimit was set at all. something like this should > >> automatically be configurable. > > > >> http://www.subethamail.org/se/archive_msg.jsp?msgId=25608 > > > >> ok. rant over. ;) > > > >> has been a rather crappy morning here. > > > >> chuck > > > > > >> Charles Williams wrote: > >>>>> almost forgot. > >>>>> > >>>>> am using the new 2.0 version of subetha. ;) > >>>>> > >>>>> why can't we have an attachment size limit setting in the subetha > config? > >> would make a bit more > >>>>> sense if you asked me. > >>>>> > >>>>> chuck > >>>>> > >>>>> Charles Williams wrote: > >>>>>> Hey guys, > >>>>>> we are still having attachment size problems. we've tried with 4, 14 > and > >> 35 MB files and all are > >>>>>> dropped with the following error: > >>>>>> Exception in thread "org.subethamail.smtp.server.Session" > >>>>>> javax.ejb.TransactionRolledbackLocalException: > >> java.lang.RuntimeException: > >>>>>> javax.mail.MessagingException: IOException; > >>>>>> nested exception is: > >>>>>> org.subethamail.common.io.LimitExceededException: Predicted > size > >> 10002432 exceeds limit 10000000 > >>>>>> at com.caucho.ejb3.xa.XAManager.commit(XAManager.java:247) > >>>>>> at > >> > _ejb.Injector.InjectorBean__EJB$Injector__EJBLocal.inject(InjectorBean__EJB.java:268) > >>>>>> at > >> org.subethamail.core.smtp.OurDeliverer.deliver(OurDeliverer.java:34) > >>>>>> at > >> org.subethamail.core.smtp.SMTPHandler$Handler.data(SMTPHandler.java:140) > >>>>>> at > >> org.subethamail.smtp.command.DataCommand.execute(DataCommand.java:55) > >>>>>> at > >> > org.subethamail.smtp.server.CommandHandler.handleCommand(CommandHandler.java:72) > >>>>>> at org.subethamail.smtp.server.Session.run(Session.java:114) > >>>>>> Caused by: com.caucho.transaction.RollbackExceptionWrapper: > >> java.lang.RuntimeException: > >>>>>> javax.mail.MessagingException: IOException; > >>>>>> nested exception is: > >>>>>> org.subethamail.common.io.LimitExceededException: Predicted > size > >> 10002432 exceeds limit 10000000 > >>>>>> at > >> com.caucho.transaction.TransactionImpl.commit(TransactionImpl.java:610) > >>>>>> at > >> > com.caucho.transaction.TransactionManagerImpl.commit(TransactionManagerImpl.java:270) > >>>>>> at > >> com.caucho.jca.UserTransactionImpl.commit(UserTransactionImpl.java:437) > >>>>>> at > >> > com.caucho.jca.UserTransactionProxy.commit(UserTransactionProxy.java:166) > >>>>>> at com.caucho.ejb3.xa.XAManager.commit(XAManager.java:243) > >>>>>> ... 6 more > >>>>>> Caused by: java.lang.RuntimeException: > javax.mail.MessagingException: > >> IOException; > >>>>>> nested exception is: > >>>>>> org.subethamail.common.io.LimitExceededException: Predicted > size > >> 10002432 exceeds limit 10000000 > >>>>>> at > >> org.subethamail.core.injector.InjectorBean.inject(InjectorBean.java:213) > >>>>>> at > >> > _ejb.Injector.InjectorBean__EJB$Injector__EJBLocal.inject(InjectorBean__EJB.java:260) > >>>>>> ... 5 more > >>>>>> Caused by: javax.mail.MessagingException: IOException; > >>>>>> nested exception is: > >>>>>> org.subethamail.common.io.LimitExceededException: Predicted > size > >> 10002432 exceeds limit 10000000 > >>>>>> at javax.mail.internet.MimeMessage.parse(MimeMessage.java:340) > >>>>>> at javax.mail.internet.MimeMessage.<init>(MimeMessage.java:209) > >>>>>> at com.sun.mail.smtp.SMTPMessage.<init>(SMTPMessage.java:109) > >>>>>> at > >> org.subethamail.common.SubEthaMessage.<init>(SubEthaMessage.java:75) > >>>>>> at > >> > org.subethamail.core.injector.InjectorBean.injectImpl(InjectorBean.java:258) > >>>>>> at > >> org.subethamail.core.injector.InjectorBean.inject(InjectorBean.java:190) > >>>>>> ... 6 more > >>>>>> Caused by: org.subethamail.common.io.LimitExceededException: > Predicted > >> size 10002432 exceeds limit > >>>>>> 10000000 > >>>>>> at > >> > org.subethamail.common.io.LimitingInputStream.thresholdReached(LimitingInputStream.java:30) > >>>>>> at > >> > org.subethamail.common.io.ThresholdingInputStream.checkThreshold(ThresholdingInputStream.java:146) > >>>>>> at > >> > org.subethamail.common.io.ThresholdingInputStream.read(ThresholdingInputStream.java:93) > >>>>>> at > java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > >>>>>> at > java.io.BufferedInputStream.read1(BufferedInputStream.java:258) > >>>>>> at > java.io.BufferedInputStream.read(BufferedInputStream.java:317) > >>>>>> at > com.sun.mail.util.ASCIIUtility.getBytes(ASCIIUtility.java:259) > >>>>>> at javax.mail.internet.MimeMessage.parse(MimeMessage.java:338) > >>>>>> ... 11 more > >>>>> > >>>>>> chuck > >>>>> _______________________________________________ > >>>>> SubEtha Users mailing list > >>>>> users@subethamail.org > >>>>> http://www.subethamail.org/se/list/users > >>>>> http://www.subethamail.org/se/archive_msg.jsp?msgId=50412 > >>>>> > >> _______________________________________________ > >> SubEtha Users mailing list > >> users@subethamail.org > >> http://www.subethamail.org/se/list/users > >> http://www.subethamail.org/se/archive_msg.jsp?msgId=50413 > > > > _______________________________________________ > > SubEtha Users mailing list > > users@subethamail.org > > http://www.subethamail.org/se/list/users > > http://www.subethamail.org/se/archive_msg.jsp?msgId=50414 > > > >> ------------------------------------------------------------------------ > > > > > >> _______________________________________________ > >> SubEtha Users mailing list > >> users@subethamail.org > >> http://www.subethamail.org/se/list/users > >> http://www.subethamail.org/se/archive_msg.jsp?msgId=50425 > > > _______________________________________________ > SubEtha Users mailing list > users@subethamail.org > http://www.subethamail.org/se/list/users > http://www.subethamail.org/se/archive_msg.jsp?msgId=50463 > > - -- > - - -- > // > // IT- & Software-Systemhaus, Internet Service und Web Agentur > // > // Charles Williams, Sys/Sec Administrator, R&D, Proj. Man. > // PLANET IC GmbH, Haus 4, Hagenower Strasse 73, 19061 Schwerin > // Registergericht Schwerin HRB 6762, Geschäftsführer Andreas Scher > // Telefon 0385 30200 200, Telefax 0385 30200 190, Internet > www.planet-ic.de > // > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAkpLDtoACgkQ0RUDxJHrYt7wvQCglwkPTK2kPKascMZxs32/sMQA > ELMAoKaJ6dlV7ohcHQluQhcqLp2Rpx5i > =K9LB > -----END PGP SIGNATURE----- > _______________________________________________ > SubEtha Users mailing list > users@subethamail.org > http://www.subethamail.org/se/list/users > http://www.subethamail.org/se/archive_msg.jsp?msgId=50464 >
Umm yea, I'd just turn off mail queue approval before doing the import and then turn it back on afterwards. =)<div><br></div><div>jon<br><br><div class="gmail_quote">On Wed, Jul 1, 2009 at 12:23 AM, Charles Williams <span dir="ltr"><<a href="mailto:c.williams@planet-ic.de">c.williams@planet-ic.de</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br> Hash: SHA1<br> <br> Jon,<br> <br> </div>so. after the change I can now import a 90+ MB archive into subetha which is great. the only problem<br> is that the archive is not imported directly into the archive in subetha. instead it is put in the<br> mail queue for approval (list holds all new messages). which means there are about 400 mails waiting<br> for approval and once approved they will be sent to all list members.<br> <br> not a very good way to import old archives.<br> <br> chuck<br> <div><div></div><div class="h5"><br> <br> Charles Williams wrote:<br> > Jon,<br> ><br> > that's what I ended up doing yesterday. bumped it to 100 MB (exim is limited to 50 MB anyway) and<br> > hopefully that will be enough. it did take forever to find the correct constant to change though.<br> > why couldn't you have answered earlier? LOL<br> ><br> > anyway. am going to try and import an archive now and see if that works now. I hope so. have been<br> > dealing with this migration for over a month now. has long since become old. :(<br> ><br> > thanks again,<br> > chuck<br> ><br> ><br> > Jon Stevens wrote:<br> >> It is odd that you are hitting this limit with a 4 meg attachment, but I can<br> >> easily see it with a larger (14/35meg) attachment.<br> ><br> >> The issue is buffering. Imagine if 100 messages come in at the same exact<br> >> time and each has a 10 meg attachment. That is a minimum of 1gig of memory<br> >> that the JVM needs to deal with. If you overflow the JVM heap size, then you<br> >> are shit out of luck. (Make sure you increase the amount of memory allocated<br> >> to the JVM... edit resin.conf and specify a much larger<br> >> <jvm-arg>-Xmx512m</jvm-arg>... I'd also specify an equally sized -Xms<br> >> variable as well.)<br> >> Agreed that this should be a more easily configured option. We can improve<br> >> on that in the future. Have you tried increasing the limit? (just recompile<br> >> subetha, it is pretty easy). Change the MAX_MESSAGE_BYTES variable in the<br> >> InjectorBean to something much larger. Here is the link to how to setup a<br> >> development environment. You can skip the eclipse part... just check the<br> >> source code out and use ant to compile a war file for you with an 'ant war'<br> >> command.<br> >> <a href="http://code.google.com/p/subetha/wiki/DeveloperGuide" target="_blank">http://code.google.com/p/subetha/wiki/DeveloperGuide</a><br> ><br> >> jon<br> ><br> ><br> >> On Mon, Jun 29, 2009 at 11:48 PM, Charles Williams<br> >> <<a href="mailto:c.williams@planet-ic.de">c.williams@planet-ic.de</a>>wrote:<br> ><br> >> well, just finished a google search regarding this and it seems to be the<br> >> same error/bug that was<br> >> first reported back in 2007. which impacts all mail whether imported into<br> >> the archives or newly sent<br> >> messages.<br> ><br> >> this is also the same problem that I reported a while back. and am<br> >> currently catching hell for here.<br> >> am wondering why a hardlimit was set at all. something like this should<br> >> automatically be configurable.<br> ><br> >> <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=25608" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=25608</a><br> ><br> >> ok. rant over. ;)<br> ><br> >> has been a rather crappy morning here.<br> ><br> >> chuck<br> ><br> ><br> >> Charles Williams wrote:<br> >>>>> almost forgot.<br> >>>>><br> >>>>> am using the new 2.0 version of subetha. ;)<br> >>>>><br> >>>>> why can't we have an attachment size limit setting in the subetha config?<br> >> would make a bit more<br> >>>>> sense if you asked me.<br> >>>>><br> >>>>> chuck<br> >>>>><br> >>>>> Charles Williams wrote:<br> >>>>>> Hey guys,<br> >>>>>> we are still having attachment size problems. we've tried with 4, 14 and<br> >> 35 MB files and all are<br> >>>>>> dropped with the following error:<br> >>>>>> Exception in thread "org.subethamail.smtp.server.Session"<br> >>>>>> javax.ejb.TransactionRolledbackLocalException:<br> >> java.lang.RuntimeException:<br> >>>>>> javax.mail.MessagingException: IOException;<br> >>>>>> nested exception is:<br> >>>>>> org.subethamail.common.io.LimitExceededException: Predicted size<br> >> 10002432 exceeds limit 10000000<br> >>>>>> at com.caucho.ejb3.xa.XAManager.commit(XAManager.java:247)<br> >>>>>> at<br> >> _ejb.Injector.InjectorBean__EJB$Injector__EJBLocal.inject(InjectorBean__EJB.java:268)<br> >>>>>> at<br> >> org.subethamail.core.smtp.OurDeliverer.deliver(OurDeliverer.java:34)<br> >>>>>> at<br> >> org.subethamail.core.smtp.SMTPHandler$Handler.data(SMTPHandler.java:140)<br> >>>>>> at<br> >> org.subethamail.smtp.command.DataCommand.execute(DataCommand.java:55)<br> >>>>>> at<br> >> org.subethamail.smtp.server.CommandHandler.handleCommand(CommandHandler.java:72)<br> >>>>>> at org.subethamail.smtp.server.Session.run(Session.java:114)<br> >>>>>> Caused by: com.caucho.transaction.RollbackExceptionWrapper:<br> >> java.lang.RuntimeException:<br> >>>>>> javax.mail.MessagingException: IOException;<br> >>>>>> nested exception is:<br> >>>>>> org.subethamail.common.io.LimitExceededException: Predicted size<br> >> 10002432 exceeds limit 10000000<br> >>>>>> at<br> >> com.caucho.transaction.TransactionImpl.commit(TransactionImpl.java:610)<br> >>>>>> at<br> >> com.caucho.transaction.TransactionManagerImpl.commit(TransactionManagerImpl.java:270)<br> >>>>>> at<br> >> com.caucho.jca.UserTransactionImpl.commit(UserTransactionImpl.java:437)<br> >>>>>> at<br> >> com.caucho.jca.UserTransactionProxy.commit(UserTransactionProxy.java:166)<br> >>>>>> at com.caucho.ejb3.xa.XAManager.commit(XAManager.java:243)<br> >>>>>> ... 6 more<br> >>>>>> Caused by: java.lang.RuntimeException: javax.mail.MessagingException:<br> >> IOException;<br> >>>>>> nested exception is:<br> >>>>>> org.subethamail.common.io.LimitExceededException: Predicted size<br> >> 10002432 exceeds limit 10000000<br> >>>>>> at<br> >> org.subethamail.core.injector.InjectorBean.inject(InjectorBean.java:213)<br> >>>>>> at<br> >> _ejb.Injector.InjectorBean__EJB$Injector__EJBLocal.inject(InjectorBean__EJB.java:260)<br> >>>>>> ... 5 more<br> >>>>>> Caused by: javax.mail.MessagingException: IOException;<br> >>>>>> nested exception is:<br> >>>>>> org.subethamail.common.io.LimitExceededException: Predicted size<br> >> 10002432 exceeds limit 10000000<br> >>>>>> at javax.mail.internet.MimeMessage.parse(MimeMessage.java:340)<br> >>>>>> at javax.mail.internet.MimeMessage.<init>(MimeMessage.java:209)<br> >>>>>> at com.sun.mail.smtp.SMTPMessage.<init>(SMTPMessage.java:109)<br> >>>>>> at<br> >> org.subethamail.common.SubEthaMessage.<init>(SubEthaMessage.java:75)<br> >>>>>> at<br> >> org.subethamail.core.injector.InjectorBean.injectImpl(InjectorBean.java:258)<br> >>>>>> at<br> >> org.subethamail.core.injector.InjectorBean.inject(InjectorBean.java:190)<br> >>>>>> ... 6 more<br> >>>>>> Caused by: org.subethamail.common.io.LimitExceededException: Predicted<br> >> size 10002432 exceeds limit<br> >>>>>> 10000000<br> >>>>>> at<br> >> org.subethamail.common.io.LimitingInputStream.thresholdReached(LimitingInputStream.java:30)<br> >>>>>> at<br> >> org.subethamail.common.io.ThresholdingInputStream.checkThreshold(ThresholdingInputStream.java:146)<br> >>>>>> at<br> >> org.subethamail.common.io.ThresholdingInputStream.read(ThresholdingInputStream.java:93)<br> >>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)<br> >>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)<br> >>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)<br> >>>>>> at com.sun.mail.util.ASCIIUtility.getBytes(ASCIIUtility.java:259)<br> >>>>>> at javax.mail.internet.MimeMessage.parse(MimeMessage.java:338)<br> >>>>>> ... 11 more<br> >>>>><br> >>>>>> chuck<br> >>>>> _______________________________________________<br> >>>>> SubEtha Users mailing list<br> >>>>> <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> >>>>> <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> >>>>> <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50412" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50412</a><br> >>>>><br> >> _______________________________________________<br> >> SubEtha Users mailing list<br> >> <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> >> <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> >> <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50413" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50413</a><br> ><br> > _______________________________________________<br> > SubEtha Users mailing list<br> > <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> > <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> > <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50414" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50414</a><br> ><br> >> ------------------------------------------------------------------------<br> ><br> ><br> >> _______________________________________________<br> >> SubEtha Users mailing list<br> >> <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> >> <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> >> <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50425" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50425</a><br> ><br> </div></div><div class="im">_______________________________________________<br> SubEtha Users mailing list<br> <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> <a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50463" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50463</a><br> <br> </div><div class="im">- --<br> - - --<br> //<br> // IT- & Software-Systemhaus, Internet Service und Web Agentur<br> //<br> // Charles Williams, Sys/Sec Administrator, R&D, Proj. Man.<br> // PLANET IC GmbH, Haus 4, Hagenower Strasse 73, 19061 Schwerin<br> // Registergericht Schwerin HRB 6762, Geschäftsführer Andreas Scher<br> // Telefon 0385 30200 200, Telefax 0385 30200 190, Internet <a href="http://www.planet-ic.de" target="_blank">www.planet-ic.de</a><br> //<br> -----BEGIN PGP SIGNATURE-----<br> Version: GnuPG v1.4.9 (GNU/Linux)<br> Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org" target="_blank">http://enigmail.mozdev.org</a><br> <br> </div>iEYEARECAAYFAkpLDtoACgkQ0RUDxJHrYt7wvQCglwkPTK2kPKascMZxs32/sMQA<br> ELMAoKaJ6dlV7ohcHQluQhcqLp2Rpx5i<br> =K9LB<br> <div class="im">-----END PGP SIGNATURE-----<br> _______________________________________________<br> SubEtha Users mailing list<br> <a href="mailto:users@subethamail.org">users@subethamail.org</a><br> <a href="http://www.subethamail.org/se/list/users" target="_blank">http://www.subethamail.org/se/list/users</a><br> </div><a href="http://www.subethamail.org/se/archive_msg.jsp?msgId=50464" target="_blank">http://www.subethamail.org/se/archive_msg.jsp?msgId=50464</a><br> </blockquote></div><br></div>